general

[loud briar]

Use --progress=logs, it's much tidier

[chrome moss]

For my own knowledge though, what is changing the way the logging behavs in CI vs local? I run in local with no settings, its nice and pretty like I want, but in CI its totally different when nothing is being passed in to tell it to do that?

[loud briar]

Your gitlab runner can't display an interactive terminal

It defaults to --progress=plain

[chrome moss]

oh man can't believe I didnt put that together... oof. Thanks for the tip though I'll def use logs instead

[loud briar]

The only 'issue' I've found with logs is that DockerBuild displays nothing. Probably something to do with hiving it off to Buildkit, but that's changing soon so 🤞 that improves in future. Otherwise it's a nice tidy output for runner logs

[cursive bridge]

Does Container.AsTarball produce a tarball with all the layers like skopeo/docker save does? I believe so, just want to verify. The description in the options implies this is the case.

[hazy rover]

Is there a way to do secret substitution from the engine instead of from the CLI? For example: dagger --some-secret env://FOO reads $FOO from the CLI's process, but I'm looking for a way to reference a secret that the engine can access but the CLI can't (could be an env var, a file, or anything - I don't care).

[cunning jolt]

This is what I've seen. although, I haven't individually verified all the layers. I use it for image scanning. It detects everything fine.

[warm temple]

Is there a way to do secret substitution

[winter linden]

yes.

[loud briar]

Think https://docs.dagger.io/features/secrets/#aws-secrets-manager-and-parameter-store is outdated? aws:// / aws:/// -> aws+sm:// aws+ps://

[warm temple]

Thanks, updating

[cursive bridge]

Running WithExec with these options:

dagger.ContainerWithExecOpts{ Expect: expectedResult, RedirectStdout: reportPath, Expand: true, },

Where reportPath = $HOME/artifacts

Results in an error open redirect stdout file: open /tmp/rootfs1507444385/$HOME/artifacts

So, it looks like Expand isn't applying to the RedirectStdout arg. Is this expected?

[winter linden]

Looks like an oversight.. Could you open an issue? 🙏

[cursive bridge]

https://github.com/dagger/dagger/issues/12039

[cursive bridge]

Is this allowed?

dagger.ContainerWithExecOpts{RedirectStdout: "/dev/stderr"}

I'm getting an error on that. Is there another way to redirect stdout to stderr?

[oak estuary]

Can use of golang sdk instead of python sdk boost up dagger pipeline speed

[winter linden]

Yes... the python sdk has performance issues, although we're working on improving it

If you want to prioritize performance, there is a new SDK with a specialized scripting language called Dang. It uses graphql types natively and therefore requires no codegen. As a result modules written in dang are very very fast

See https://dagger.io/changelog/#dang-a-dagger-native-language

[tired moth]

Is this going to be officially supported because dang repo is not in dagger org in github https://github.com/vito/dang

should we start migrating to dang over ts sdk internally? is it prod ready as v1?

@elfin frigate can you please help here for dang direction?

[elfin frigate]

no, it is not prod ready

it's highly experimental and will almost certainly have backwards incompatible changes from version to version, at the moment you're expected to just learn from examples in our repo and dang's

[tired moth]

ok thanks for clarifying, so if we have written dagger code in ts then eventually we may need to migrate it to dang in case if we want to use dang then, right?

[elfin frigate]

well, you will never need to migrate, but if you want to use dang, then yes 😛

[tired moth]

ok cool thanks.

[loud briar]

I see a bunch of open MRs on workspaces, what's happening there?

[jagged flicker]

Workspaces are a new (better) way to deal with modules. This is also called "Modules v2". This is this changelog item: https://dagger.io/changelog/#modules-v2 (with links to the design docs)
Workspaces is landing step by step in the main line, some parts of the workspace api are already there.
One of the main issue solved by workspaces is to clearly make the distinction between developing a module and using a module

[loud briar]

Yeah I'm quite caught up, have already sold the idea internally, just not seeing much movement on those MRs. Quite an anticipated feature for us because we have a dividing line between maintainers and users

[winter linden]

we've been splitting it up to get it merged faster. First chunk was workspace API (already merged https://github.com/dagger/dagger/pull/11874). Second chunk is workspace plumbing (https://github.com/dagger/dagger/pull/11995). Another small chunk dropping today (for lockfile support). The remaining code (mostly UX) will be rebased and shipped in https://github.com/dagger/dagger/pull/11812

[loud briar]

Ah ha. I did wonder who'd review that 300+ commit change PR 😂

[cursive bridge]

Large PRs get merged the fastest with the least issues....because no one reviews them

[tired moth]

Is this workspace feature released? if yes then docs updated for it or not? Please help

[winter linden]

My friend as I already told you: you can check https://dagger.io/changelog

[tired moth]

opps sorry. crazy fan of dagger forgot this that i am already checking in changelog in devlopment phase only. sorry

[winter linden]

We're going to ship it soon I promise 🙂

Appreciate the passion

[tired moth]

I only wish for one feature from the last 4 years, live development support, please ship that also soon, me and many devlopers will give so many blessign to you for that. really. thanks for everything

Workspace feature is similar like npm workspace if yes then we will be able to ditch npm workspace and migrate to dagger workspace? just curious so asking

[cerulean token]

I know that there is a blog about the transition from Earthly to Dagger: https://dagger.io/blog/earthly-to-dagger-migration/. Are there any other transition case studies/examples? Specifically, I'm curious about Bazel.

Also, I've recently been using Dagger with Godot (https://godotengine.org/) for a hobby game. It works very nicely. I've got some modules that I'd like to get onto Daggerverse once they've settled.

Gotta give some credit to Godot for having a decent enough CLI interface that it can be ran in a container, too 🙂

[elfin frigate]

subscribe. I've also been learning Godot, it's pretty nice!

[hidden dirge]

I'm finding adding custom opentel spans via user dagger modules a bit hit-and-miss in terms of reliability. I know documentation is also fairly scant for this at the moment also.

Couple of questions:

1. Does the instrumenting module name need to have a particular value, maybe related to the main module name / object?
2. Is it possible to hide the module init spans somehow?
3. Specifically for the python opentelemetry sdk, does the decorator form of Tracer.start_as_current_span work? I've done some experimenting but haven't a hard time telling what's failing where.

Ultimately trying to move towards traces more like those [described above ](#general message)

[prisma imp]

what happened to container-use?

[winter linden]

The Dagger team is no longer investing in it (we are refocusing on CI and deterministic test execution, and the two engineers who started CU have left). But @white zephyr expressed interest in maintaining it, so we gave him access. See #container-use if you want to follow or participate.

[cursive bridge]

Silly question, where is the dockerfile used to build the dagger-engine containers?

[jagged flicker]

There is no Dockerfile. The function building the engine container is here: https://github.com/dagger/dagger/blob/958d5e559ccf0fcaf67af39fd6609368fb6105bd/toolchains/engine-dev/main.go#L142

[cursive bridge]

Ah, I thought dagger might be using itself to build. So, what initiates that process? IE, if I want to build the container image for dagger-engine, what would the process be? I assume there's a file somewhere I can follow for the whole build/release process?

[torpid maple]

Anyone found a good way to perform the OAuth authorization code from a Dagger function?

e.g. it opens the browser has the user authenticate then redirects them back to localhost whereby a local process listening on a open port captures the code an exchanges it for an access token?

[jagged flicker]

I thought dagger might be using itself to build
Yes, that's the case, dagger is using dagger to build.
if I want to build the container image for dagger-engine, what would the process be?
I think it depends what you want to achieve.
If you want to just build an image, from a clone of dagger/dagger, you can just run dagger call engine-dev container. It will build an return a container.
If you want to export it to your host, then you can do dagger call engine-dev container export-image --name "my-dagger-engine:latest" for instance.
That said, most of the time we are using either the scripts under hack to build a dev engine/use a dev engine or dagger call engine-dev playground terminal that will allow to enter a running dev engine.
Regarding the release process, it's inside the release toolchain. For instance this publish function: https://github.com/dagger/dagger/blob/958d5e559ccf0fcaf67af39fd6609368fb6105bd/toolchains/release/main.go#L103

[winter linden]

To build an engine container directly from a remote git commit or ref of your choice:

dagger -m github.com/dagger/dagger@v0.20.3 call engine-dev container

then you can chain all the follow-up functions that @jagged flicker mentioned

also works branches and pull requests 🙂

[cursive bridge]

TY both! I'll dig into that logic. We need to rebuild the engine container for our use, so want to make sure we mirror what dagger actually does.

[winter linden]

do you need to customize it in some way? There may already be a parameter to the build function that can help (or we can add one)

[cursive bridge]

No, it's a security thing. It's so chainguard can build and provide the image.

[winter linden]

I see. how deep does it need to be integrated? would you configure the custom build, or would it be provided to you by CG? (or do you work at cg?) would it be hosted on their infra or on yours?

[cursive bridge]

CG would just build the dagger-engine container, we would be consuming their build/container vs upstream one. It allows them to stay up to date with any security vulnerabilities and remediations.

[cunning jolt]

Doesn't dagger also use chainguard (wolfi) to build the image?

[winter linden]

custom build

[winter linden]

we use wolfi but I don't think what this is about. discussing in #1486028660896043130

[cursive bridge]

Random note, I'm hiring for a position and I have actually seen dagger pop up in a few resumes. Not a ton, but for the sample size not bad. Certainly more than I was expecting. It's nice to see.

[winter linden]

Nice ! Good to know. Hopefully with the upcoming improvements, that considerably flatten the learning curve, we will see even more of that.

One benefit of the steep learning curve: someone who was skilled and motivated enough to master Dagger, is problably an above average candidate 🙂

[cursive bridge]

Or they added a buzz word to their resume. Always hard to tell until you talk with the candidate. 😄

[cunning jolt]

Also an indication of adoption if "Dagger" is a buzz word 🙂

[loud briar]

One selling point I've used internally is that Claude (and other agents) can do the full cycle on your laptop now - edit the module(s), run the pipeline, read the results, iterate. Candidates don't need to be pros, but they do need to intuitively see that a fully local, reproducible feedback cycle is naturally going to be better for the agentic future than bash-in-yaml-in-gitlab 😉

[winter linden]

Yup, official Dagger skill coming soon that will supercharge this

[loud briar]

All about that life, Claude does more work than I do now

[tired moth]

@winter linden as you are working on issue refinements then why can't we use github issue types across dagger org in github for more visibility in diff repos in dagger org https://docs.github.com/en/issues/tracking-your-work-with-issues/using-issues/managing-issue-types-in-an-organization

[steel sail]

Why did you convert a bunch of my issues to discussions?

[winter linden]

For better categorization, issues were piling up and becoming unmanageable, we're going to follow Mitchell Hashimoto's lead on ghostty, and separate user communication (help, requests, contributor discussions) and pure work tracking

This complain triggered it: https://github.com/dagger/dagger/discussions/12050

[steel sail]

I think this is going to massively hurt discoverability of existing issues.

So the original complaint is that the OP is daunted by the number of outstanding issues. And your solution is to convert them to discussion? Some of my reported issues were not a discussion but potential bug reports. This to me feels like an attempt to conceal legit issues for better optics.

[winter linden]

I wouldn't say it a solution, but an attempt at an improvement. It's not simply converting them to discussion:

1. We closed a bunch of stale issues
2. Re-organized discussion categories
3. Individually classified each new discussion between "Ideas", "Help", "Maintainers" and general. "Help" being where bug reports and support requests go - with the goal of being more visible. Not a very high bar given the current state of issues.

The big win IMO is that many of those issues were open-ended discussions that drowned out 1) genuine bug reports, and 2) users asking for help. This conversion/classification, I hope, will help elevate those.

It's not a silver bullet, we still have to invest the time to answer and prioritize those requests... But that was already true before, and we were dropping the ball on many of those. I'm hoping that this helps us drop the ball less.

One way or the other we had to do something because that pile was just growing and growing

[steel sail]

How’s this a discussion?

https://github.com/dagger/dagger/discussions/12816

This is an issue that you have confirmed to be real and you’ve even released a fix for it.

[winter linden]

Yes this obviously deserves an issue. The natural flow would be: 1) user reports an issue; 2) in some cases, maintainers create an issue to track work.

Don't take it as a demotion of the importance of that work, or of your report. But look at the other side of this: the vast majority of these newly converted discussions should not lead to an issue being created. And it was a massive tax on us that they were.

The root cause is Github issue's dual purpose as a task tracker, and a support desk

Now we have a clean slate to create (or in this case, re-create) issues that are truly tracking engineering work planned by the maintainers

[steel sail]

Ok, this is obviously your project so you should manage however you see fit. But I will politely disagree and commit.

[winter linden]

If there's a better way you think we can handle issue rot / sprawl, I'm interested. And will be glad to be proven wrong if the result is something better 🙂

And, it's easy to revisit later if this turns out to be a mistake. The data is not lost

[steel sail]

I don’t know the exact definitions when you say of “issue rot” and “sprawl”?

[winter linden]

• issue rot: issues stay opened too long, don't get attention, questions go unanswered, nobody knows whether to close or not, etc
• issue sprawl: too many goddamn issues, poor signal/noise ratio, hard to organize and prioritize, etc

[tired moth]

@winter linden Any consideration?

[winter linden]

I had never heard of this feature... But honestly I've been burned by Github's "advanced" project management features, and I don't really trust them beyond the basics

[tired moth]

We are using this feature since its availability and so its very helpful for teams, maintainer and community also. Just check it once. Thanks

[winter linden]

For reference: https://x.com/mitchellh/status/1852039089547309552

Mitchell Hashimoto (@mitchellh)

I'm doing something new (for me) with Ghostty: the issue tracker is only for actionable tasks. Features in the issue tracker are accepted and well-scoped. Bugs are reproducible. PRs must have an associated issue (no drive-by PRs). The issue tracker is not used for discussion.

X · Oct 31, 2024, 5:25 PM

[stoic knot]

Wanted to say thanks for putting up with my proxied complaints and working to make Dagger better! Yesterday, I saw an unsolicited comment in slack noting recent improvements to perf and stability.

[heavy garnet]

Hello, I hope you're fine !
As I've worked a bit on Dagger those last months and still intend to keep working on it, I redacted a set of blog articles which acts as a "Deep Dive" in 4 parts. Most of you probably won't learn new things about it, but it's a recap/compilation of everything I learned along my journey, and my personal experience with the tool (which was great so far).
I intend it mainly for my coworkers and people around, but if it can help the community, I'll be glad !

https://dev.to/sami_chibani/cicd-in-the-era-of-ai-and-platform-engineering-a-deep-dive-into-dagger-ci-part-1-58pi

P.S: Thanks again for the work you're doing on this project, for someone following it since 2022, I really enjoyed the journey (and the direction it takes)

[winter linden]

Thanks for sharing! And thanks for the support 🙂

[torpid maple]

Still wondering whether this is viable with Dagger or will I have to wrap the Dagger call in some bash script that handles this part first.

That would feel like a shame as I see the whole point of using Dagger to assist with making things portable and independent of execution environment.

If I have to wrap Dagger in a bash script and add a bunch of defensive checks to it in order to handle multiple platforms correctly, then why bother using Dagger at all?

Ideally I wanted to do dagger call auth and it would check if a cached token was valid and can be used, if not it would move forward with performing the OAuth authorization code flow to authenticate the user which involves going to a URL logged by the CLI, then once the authentcate has been completed in the browser it redirects back to localhost:<port> whereby the process obtains the authorization code and does the token exchange.

But it doesn't appear to be possible to have Dagger expose a service port outside of the running dagger call <function> up, which is not quite my use case.

[sharp marsh]

@torpid maple do you need the auth flow to be part of a larger pipeline where the step hangs waiting for the user to visit a URL so the proper tokens get passed all over?

[torpid maple]

So the idea is, you run the same Dagger function both in a CI pipeline and locally as a developer.

Depending on the context (whether CI env var is set), it will either connect directly to Vault using the IDP that the CI environment has access to, or it'll prompt the developer to authenticate as themselves.

e.g.

$ dagger call do-the-thing      # -> used cached token/prompt developer to authenticate -> do-the-thing
$ dagger call do-the-thing --ci # -> authenticates with service account in ci context   -> do-the-thing 

[winter linden]

3 options off the top of my head:

1. Stopgap: use an external tunneling service (ngrok, cloudflare warp, tailscale tunnel) and orchestrate it from your dagger function

2. We add Workspace.forwardPort() and let modules call it at will . (bonus we could make the same choice of tunnel providers swap-in CLI integrations 🙂

3. maybe leverage upcoming dagger up and make the oauth tunnel a long-running service in your stack? could that work @jagged flicker ?

[sharp marsh]

@torpid maple is it a requirement for the oauth tool to live inside Dagger? Another option is use the cmd secret provider so the auth happens in the devs machine effectively. Something like dagger call foo --token="cmd://gh auth token"

[torpid maple]

That's an interesting option I wasn't aware of, but I assume the developer would have to know what command to specify and it couldn't be set as a default arg?

[winter linden]

the module cannot make it a default from inside its sandbox, but you can configure it in your project- with .env today and new workspace config file soon

[sharp marsh]

this. User defaults should work https://docs.dagger.io/features/local-defaults/#supported-values

[torpid maple]

Of those options which is possible right now, only option 1 right?

[winter linden]

Option 1 and 4 (cmd://)

[torpid maple]

I don't think option 1 would work because the redirect uri usually has to be configured and whitelisted in the identity provider and at the moment only localhost would be allowed.

The URI generated from those tunneling tools is often dynamic so wouldn't be able to whitelist it in advance

So I guess that leaves me with only option 4 as viable.

Which is also a shame as it essentially means I can't do any caching or setup of the tool they need to perform that command as part of dagger

but oh well 🤷‍♂️

[winter linden]

@torpid maple another option would be to run a local tunnel broker - a server that backend workers can connect to and register routes - the browsers connect to that route and the broker just proxies. Would be trivial to vibe-code, but there's got to be a stable lightweight OSS implementation of this?

[sharp marsh]

the caching will work with cmd as long as the token returned is the same. Not sure how your oauth cli app handles that

[torpid maple]

I'll investigate and report back, thanks for the suggestions it's much appreciated!

[winter linden]

@torpid maple my takeaway:

• Plan A: if specialized auth CLI available: run it withcmd:// for example cmd://gh auth token

• Plan B: if no specialized auth CLI available: run socat on the host with 2 TCP listeners 🙂 socat TCP-LISTEN:8080,reuseaddr TCP-LISTEN:9090,reuseaddr. Have your dagger function connect to one, and redirect the browser to the other.

Both work today

[cunning jolt]

@torpid maple would this help? https://github.com/dagger/dagger/pull/11929

[cursive bridge]

I'm stuck on a weird error. I'm running dagger engine in kubernetes. I've been doing this in a different context for testing and am trying to deploy for our general use. The dagger cli is being called from a gitlab runner job pod, and the logs show this error:

Error: failed to check if module already exists: failed to stat local path: failed to stat path: failed to get requester session: session for "qoail8ophjzqgl29tjt9s92ye" not found

[winter linden]

@steel sail update: I did a cleanup pass and re-opened 15 issues that were incorrectly converted to a discussion.

While I was at it, opened 12 PRs to fix most of them 🙂

We also setup automations internally to notify us whenever there's movement on Help discussions, and we'll keep a higher bar for issue triage. I'm still hopeful that the result will be a better support experience for the community, less maintainer burnout, and faster improvement.

Thanks for caring

#maintainers message

[sharp marsh]

@torpid maple the oauth case kept wandering in my head. Do you have a way to set your CLI to listen to a specific port for the localhost oauth callback? If that's the case there might be a way where you can make this work fully in dagger for the local workflow by splitting it in two calls.

dagger call auth would spin up the oauth CLI as a service and then store the token in a cache volume. Then you can just call any other dagger pipeline and mount the same cache volume the auth call used to get the secrets

LMK if that makes sense

[chrome moss]

I wanted to ask/spark discussion about what is the biggest hurdle to cross on moving to a toolchain focused set up using constructor args where possible to condense what is required for a user to write dagger code for. We have a scenario where we have written small "core" modules I call them (python linters, yamllint, container builds, etc), that are then pulled into a toolchain module "pipeline". From here, we can then create checks and use constructor args for users to set optional args to these checks. Users then add the toolchain module to their dagger.json, and are almost at a point where no code is required, they run dagger checks and they are off to the races for the most part. This is awesome, works great.

The breakdown of this is when a user requires options that are not easily captured in a constructor and is not static. Private auth tokens being the main example here. One user might need 2 tokens, another 3, etc. For python, each token requires a username and registry url to handle properly. Trying to handle this is ugly, leading to args like indexRegistry1 string indexName1 string, indexPassword1 dagger.Secret , up to a number we arbitrarily handle in the constructor. You could marginally make this better by setting arrays, but you can't set an array of dagger.Secrets in an .env file, and even if you could, this is pretty brittle/cumbersome to manage and error-prone on the user side.

I am not sure if workspaces will help alleviate this or not, but I thought I'd mention it and see if you had any thoughts/discussions about it

[sharp marsh]

If the pipeline has potentially dynamic inputs wouldn't it be better to setup your modules in a way that those dynamic values are resolved within the module somehow?

For example, your users could submit one token to access a secret manager from some sort (1p, vault, ssm, etc) so the dynamic secrets get resolved in runtime when the module is called

[chrome moss]

I am not seeing how this gets you out of the issue. That would still require users to need varying commands being ran based on tokens they have in their vault correct? Let me use an example that could apply that doesn't handle creds for example.

I have a python module with various linters. Each linter always starts with a base Python() container that takes a source, and runs a uv sync. A user can call Mypy, it will build the base Python container with source, and run UV Sync if needed for dependencies. That way, no matter what linter is called the same base container is being used(cached) to run the linter. From here, we have a builder function for the base container to handle one-off scenarios. WithEnvVariable(). If a user wants to run Ruff but needs some other env, they would say dagger call python with-env-var ruff that then adds their var to the base container before running Ruff. This works fine in a scenario where a user is assumed to be running/building their pipelines locally in their .dagger because they can substantiate their base first in their main.py, and then write out all the related linting functions from there.

But in a toolchain scenario, this presents a problem. If I set all my python linters as checks, have a user install it as a toolchain, then they get those checks for free. They basically just set PYTHON_SRC in their dagger.json, and run dagger check and away they go. Great! But what if a user needs an environment variable for their checks to run? Now we have 2 choices I see: 1) Try and handle this in an arg in the constructor of python, which leads to what I described with secrets above, or 2) They substantiate their base first in a local .dagger module, then need to restate the provided python linters and add checks to them, rendering the point of the toolchain moot.

[patent slate]

I'd say, you are still stuck in the "give users config to set things up" mindset. How about "give users the ability to code to set things up"? This is a weak, but hopefully paradigm-shifting example.

Tool chain module:

@object_type
class PythonToolchain:
    def with_registry(self, url: str, auth: Secret) -> "PythonToolchain":
        # logic to add credentials to the internal state
        return self

    @function
    def linter_check(self) -> Container:
        # runs the actual logic

User's Local Module:

@function
def CI(self):
    return (dag.python_toolchain()
            .with_registry("ghcr.io", dag.set_secret("a", "b"))
            .with_registry("docker.io", dag.set_secret("c", "d"))
            .linter_check())

The magic is in method chaining and allowing the users, (who are devs right?) to control their own destiny. Code via Dagger (and not config) allows exactly for this inversion of control. You build the CI framework, the devs make it work for their purpose.

[cunning jolt]

Toolchain DX

[cursive bridge]

I'm looking at configuring custom CA for the engine, is this doc up to date?
https://docs.dagger.io/reference/configuration/custom-ca

Anything in the logs to indicate the engine finds and loads the certs?

[cerulean token]

iirc, the entrypoint of the dagger engine container copies those certs to the right location for the OS then runs the equivalent of update-ca-certificates for that OS. You should see the output of that script at the beginning of the logs

note that this only happens are startup (again, iirc)

EDIT: code ref https://github.com/dagger/dagger/blob/main/cmd/engine/main.go

[cunning jolt]

This is correct. If you search for update-ca-certificates near startup of the engine logs you will see the certs being pulled in. I bundle them in my own engine container.

[cursive bridge]

Hrm, I do not see any logs mentioning update-ca-certificates. Is the location for the cert files correct in the docs? (/root/.config/dagger.ca-certificates)

[cerulean token]

The link that I edited into my message implies that the update-ca-certificates exec output gets swallowed unless it errors out, so it may be succeeding silently

[cunning jolt]

That may be the case now. I tested this in early days of cacert support.

[cerulean token]

yeah I vaguely recall seeing output as well when I first did it

Can't check the blame easily, I'm on mobile. Sorry I can't be more help, hope that gets you moving in the right direction!

[sharp marsh]

@cursive bridge do you have the path correctly? Certs should be in ~/.config/dagger/ca-certificates/my-cert.crt

[sharp marsh]

here's how we're currently testing this https://github.com/dagger/dagger/blob/ae2491bff148ad49a0eb40918c9d44389120da01/core/integration/provision_test.go#L127-L169

take into account that the file needs to be there before the engine starts

[cursive bridge]

I'm running the upstream engine containers on kubernetes. I'm mounting a secret with the certs into the pod, and since I can't do $HOME relative paths I'm mounting in /root/.config/dagger/ca-certificates/, which should be correct according to the docs and that the container looks to be running as root

[sharp marsh]

if you exec into the engine pod, you should see the certificate in /usr/local/share/ca-certificates

oh wait, if you're using k8s that's not going to work

because you're actually provisioning the engine

when you're provisioning the engine yourself, you need to mount the certs in /usr/local/share/ca-certificates in the engine pod. It's described here:

[cursive bridge]

I saw that, and was kinda confused which path to take. Just below that there's a paragraph that says:

Alternatively, if you are using the default Dagger engine (e.g. via dagger call or the SDKs), you can place your custom CA certificates in the ~/.config/dagger/ca-certificates directory (or $XDG_CONFIG_HOME/dagger/ca-certificates if that environment variable is set). Dagger will automatically detect and install them on the engine.

[sharp marsh]

makes sense. It's not clear that this talks about self-provisioning. Fixing

[cursive bridge]

So, I should mount in /usr/local instead of ~/config, right?

[sharp marsh]

in the engine, yes

[sharp marsh]

https://github.com/dagger/dagger/pull/12882

[cursive bridge]

Does the engine copy these certs into running containers? IIUC, the docs say that only occurs if the base image is from a few disributions. Is there a way to ensure the containers get those certs regardless of base image distro?

[sharp marsh]

there's no way to guarantee it'll work in all distributions because there's no "standard" way of doing this. More info here: https://github.com/dagger/dagger/commit/5e933ab7f0f1dd01257993443eae1f8ce0eaa543

[cunning jolt]

I still add cacerts to downstream containers when needed

[cursive bridge]

I'm still getting cert errors. The codegen is having issues with proxy.golang.org, which is odd.

✘ withExec codegen generate-module --output /src --module-source-path /src/anchore/tests --module-name tests --introspection-json-path /schema.json --lib-version v0.19.11 1.6s ERROR
...
go: dagger.io/dagger@v0.19.11: Get "https://proxy.golang.org/dagger.io/dagger/@v/v0.19.11.mod": tls: failed to verify certificate: x509: certificate is not valid for any names, but wanted to match proxy.golang.org
2026/03/31 15:18:37 ERROR post-command failed error="exit status 1"
Error: exit status 1

That is coming from a dagger install .. for test modules to install the latest module to test against. I'm not sure where the certs should be for this operation. The engine should have all relevant certs, but the container the pipeline would run in may not. What container/engine part is doing the codegen?

[cerulean token]

wow! I didn't know Dagger did this. I can't believe that you guys did this at all. CA trust and proxy variably propagation are the only two headaches that remain for me when using containers. I've always wanted to write a controller that does this for Kubernetes Pods. Such a nice flourish!

[sharp marsh]

@cursive bridge if you exec into the engine container do you see the certificate there?

[cursive bridge]

Yes, I see the ones I added as well as the ones that come with the dagger engine container (different locations of course).

[sharp marsh]

if you see the one that you added then it should work

[cursive bridge]

The complaint is around proxy.golang.org which I would expect is already covered by certs dagger provides. We do use istio, but it isn't obvious that there is an intercept problem to me since the error is still complaining about proxy.golang.org and not some other cert.

[sharp marsh]

The engine codegen which uses proxy.golang.org is running in a container which should have your custom certificate

do you have to set a custom proxy URL in your org for traffic to flow through?

if you run dagger -M -c 'container | from golang:1.26 | terminal' you should be able to see that the container has the correct certificate located in /etc/ssl/certs

[cursive bridge]

No. This all works fine w/o istio involved, so it's clearly istio related somehow.

[cursive bridge]

I can exec into the dagger-engine pod and see the certs in /etc/ssl/certs and the custom ones I mount in in /usr/local/share/ca-certificates/. I'm taking it on faith that the engine is loading those certs though since the logs don't indicate anything.

[sharp marsh]

@cursive bridge you don't see any update-ca-certificates in the engine logs when you run dagger develop in your module?

I'm testing this right now with a custom certs and it works as intended

[cursive bridge]

Oh, I do see a message about update-ca-certificates. I looked earlier and I didn't see anything in startup. So it must only occur when something runs in the engine?

Does an ENV need to be set or anything to point at the custom certs?

[sharp marsh]

that occurs whenever you run any with-exec in a container. If update-ca-certificates is avaiable, it runs as a pre-run hook to set up the certificates accordingly

[sharp marsh]

nope, they're installed automatically

[cerulean token]

question: if I mount my ~/.azure or ~/.aws via withMountDirectory into a container, would we expect the az and aws CLIs to pick up the session? Follow up: if I refresh my token while the engine is running a function, would that change in ~/.aws or ~/.azure make it into the container via the mounted directory?

[sharp marsh]

via withMountDirectory into a container, would we expect the az and aws CLIs to pick up the session?

yes. Make sure the owner of the mounts is according to whatever user is set in the container

if I refresh my token while the engine is running a function, would that change in ~/.aws or ~/.azure make it into the container via the mounted directory?

no, withMountDirectory mounts the files from the build context which happens the moment that you run dagger call. It's not a bind-mount from your local machine

[cerulean token]

OK, one more follow-up: is the difference between withDirectory and withMountDirectory that the latter wouldn't get persisted into the final image if you published the container?

[sharp marsh]

exactly

[tired moth]

can we configure lxc as dagger container engine?

[sharp marsh]

#general message

[tired moth]

but i have read somewhere dagger is based on linux container system and using it somewhere, is it right? so configuring this can not help to save resources and run things faster?

[sharp marsh]

no, not really. It's not our focus to support all available possible container runtimes. PRs are welcome though if you wish to give it a try

[tired moth]

Will you help to review and get it right and guide for it?

[sharp marsh]

can't promise anything, sorry. It's not a priority for us. We'll try to help given that we have the time to do it

[tired moth]

alright thanks

@sharp marsh Distributed caching & build is going to be supported apart from dagger cloud also?

[sharp marsh]

yes, build is already in private beta. We have a few customers using it. Happy to give you a demo if that's a service you might be thinking of subscribing to

[tired moth]

but that is though dagger cloud only right? can we use on premise?

[winter linden]

Not supported at the moment, but I'm curious: what's your use case for on premise?

[tired moth]

to configure and maintain k8 to scale dev & ci servers is tedious if something is helping companies to replace it with smaller but powerful technology like dagger distributed build then it can help teams a lot.
Also something which can replace docker swarm or similar kind of technology.
I had discussion about this in aug 2023 with jeremy

[winter linden]

Sounds like a pitch for Dagger Cloud?

[tired moth]

haha yes, any hope or considerations?

[winter linden]

Yes, if configuring and maintaining k8s to scale dev & ci servers is tedious, I recommend trying Dagger Cloud which does all of that automatically for you

[tired moth]

but to try it we need team plan right?

[winter linden]

Yes, it's not free if that's what you mean. You have to buy the compute from somewhere 🙂 If you buy it from us, you support the development of the project (and we certainly can't afford to give it away for free)

[tired moth]

alright, cool

why cant we have $2 or some base plan to give dagger premium features like docker and not everything is free...
like community version and pro version or somehing?

@sharp marsh rather then lxc support, incus would be better right? https://github.com/lxc/incus

@tired moth @sharp marsh if we want to give pr for live devlopment support then which was your last work we should take as reference? this one https://gist.github.com/jedevc/96897a03ea54223351231752eca934f2

[cursive bridge]

FYI, Finally figured out the dagger-engine and istio issue. If anyone else is running into cert issues when using istio, try adding this annotation to your stateful/daemon set in the pod template:

traffic.sidecar.istio.io/includeInboundPorts: "<port where engine is listening>"

[sharp marsh]

happy that you sorted that out and sorry that you have to deal with Istio 😛

[cursive bridge]

We all deal with unique situations from time to time. 😄 Hope that info is helpful for someone else

[cerulean token]

wait does istio have a stigma??

[winter linden]

Not at all! (gif unrelated)

https://giphy.com/gifs/8vOEIc1tadLs25Ext5

[cerulean token]

Oh, ok great! (gif unrelated)

https://klipy.com/gifs/cheers-cheers-to-that-1

[tired moth]

Please help

[winter linden]

My friend you know we appreciate your support and energy. But please chill.

[tired moth]

haha right but dagger is really great technology bro, i just want to push to next level without interrupting main maintainer flow
You can check work in progress here in which we have integrated support for linux incus https://linuxcontainers.org/incus/introduction/ see here https://github.com/StaytunedLLP/dagger/pull/4

[winter linden]

❤️ I'll have something for you soon for live dev 🙂 Wrote a prototype

Just need to merge get a few other PRs merged first

[tired moth]

Woho...Great news...you made our whole org's day today...thanks a lot...

[cursive bridge]

I think we just hit this issue:
https://github.com/dagger/dagger/discussions/12186

What should the value for defaultPath be? I thought it was always relative to where the module runs on the host (ie . would be the CWD on the host where the module is executed), but it seems we are seeing it is local to the dagger module (at least when using call -m).

[cunning jolt]

Are you using your module as a toolchain?

[cursive bridge]

No. We just use -m for a repo with a module or local dir with modules

[loud briar]

This behaviour is changing with modules v2 in the near future I believe, though possibly not for -m. Modules installed in the workspace will have access to that workspace via a *dagger.Workspace argument

[cunning jolt]

Right, defaultPath is going away AFAIK. However, -m with defaultPath should work currently. I had issues when using toolchains but not regular modules

[loud briar]

I don't think that's quite right. The context for a remote module is the module, not the directory you call it from. That's a security/sandbox thing

[cursive bridge]

This explains it. I did development/testing with modules locally and didn't run into this issue. Once we had a daggerverse repo up and used -m on a module in that repo we hit this issue.

[cunning jolt]

Oh, yes. I'm thinking of toolchains/blueprint

[cursive bridge]

I would highlight that this differentiation isn't obvious. Probably why that issue was filed (and turned into a discussion).

[cunning jolt]

I agree. I think that's part of why workspacesV2 is being worked on. Reduce confusion on "what runs where".

[cursive bridge]

Actually, it seems we're hitting this even with a local module. Not sure why I didn't run into this before.

[loud briar]

Modules v2 will make this a much better user experience I think, and solve that discussion. For now you'll need to pass your source directory(ies) as arguments

[loud briar]

-m <a module not in this git context> will always have the module as context directory for resolving defaultPath arguments

[winter linden]

@cursive bridge @loud briar yes that's right. We are disambiguating this finally.

If you want to access the caller's workspace from a module (including when loaded by -m) you can use the new workspace API.

Declare an argument of type Workspace, it gets injected automatically - then query it for dynamic filesystem access.

--> this is already released.

[loud briar]

I didn't realise this was already working, I thought a module had to be installed in the upcoming .dagger/config.toml or as a toolchain to be able to access the workspace.

Isn't there a slight sandbox leak there, a remote module called by -m can scoop up your directory

[winter linden]

yes we are harmonizing -m with the new model. So a module is a module no matter how you load it - simpler and still backwards compatible 🙂

Re sandbox: that's the tradeoff, modules get more access - but workspace itself is a sandbox. Also you can decide to run in an empty workspace if you don't trust the module.

[loud briar]

Not an issue for me, we're only using private modules anyway, just an interesting shift of the sandbox boundary

[cerulean token]

👋 I'm having trouble with a particularly nested scenario that I'm running into with Dagger (trying to follow the testcontainers pattern):

• Running go test inside of my project's Dagger module, sindri-dev
• From within that go test, launching a subprocess that changes the working directory to a directory with a different Dagger module, sindri
• That subprocess calls dagger.io/dagger.Connect
• Then trying to use the sindri module results in:

returned error 422: {\"data\":null,\"errors\":[{\"message\":\"Cannot query fi
eld \\\"sindri\\\" on type \\\"Query\\\". Did you mean \\\"sindriDev\\\"?\",\"locations\":[{\"line\":1,\"column\":7}],\"extensio
ns\":{\"code\":\"GRAPHQL_VALIDATION_FAILED\"}}]}

It seems that the Dagger-in-Dagger client doesn't load its working directory's module into the engine like the dagger CLI does. Hoping someone can gut check my conclusion here.

[winter linden]

which versions of inner & outer dagger?

[cerulean token]

both 0.20.1

I can put together a more simple repro to make sure it's not my fault if this doesn't ring any obvious bells for you folks 🙂

alternatively I might just pin it until workspaces

[winter linden]

yes please

[winter linden]

workspace internals are already merged (just not exposed in the UI). Maybe try on main?

[winter linden]

MERGED: dagger up. Bring up all the services in your Dagger workspace. Like docker-compose up, but programmable 🙂

https://dagger.io/changelog/#dagger-up

[tired moth]

As world is moving towards agent sandbox we should rename container-use to sandbox feature of dagger because it was far ahead of time, great visionary team of dagger, congo congo...
As such world is now moving towards agent sandboxing only

[tired moth]

with dagger up we will able to expose dagger engine to local network like docker one?

[winter linden]

dagger up will expose your project's services on the local network, like docker, yes.

But better than docker, because even if you run the services on a remote dagger engine, they will still be reachable on your local machine. Dagger tunnels the forwarded ports itself as part of its API - docker does not.

[tired moth]

And dagger engine can be exposed in local network?

[winter linden]

You can already configure the dagger engine to listen on a local address if you want. But, the right configuration depends on what you want to do exactly... In general you have to be careful because the dagger engine has privileged access to its system, and it does not authenticate access. It's up to you to lock down access.

(This is not related to dagger up. dagger up is not for exposing the dagger engine itself on the network)

[tribal bison]

i feel stupid but. how can i get logs and results from dagger check cleanly? i just want to know for each check if it passed or failed + if failed i want to see the logs for that specific check

-# where "I" is not really I but an ai agent lol

[viral garden]

Hey, really impressed with Dagger so far! Quick question: When running Dagger workflows “as a service” style behind an API, would you recommend using the SDK, or packaging them up as modules? Can I combine modules and the Dagger Go SDK whist still retaining type safety, or should I just ditch modules altogether. What’s the recommended approach? Thanks so much!

[cerulean token]

I'd recommend checking out Dagger's ability to generate a Go client for a module. You can then use your module much in the same way that you can use Dagger's Go SDK

[viral garden]

I'd recommend checking out Dagger's

[sharp marsh]

hi Ema, running dagger --progress logs check should do the trick

[tribal bison]

hi Ema, running `dagger --progress logs

[winter linden]

New blog post explaining cache control for modules (function caching)

--> https://dagger.io/blog/cache-control-for-modules/

[winter linden]

The X post if you feel like boosting: https://x.com/dagger_io/status/2041207631675793609

Dagger (@dagger_io)

Introducing cache control for Dagger modules.

Dagger executes your CI pipelines incrementally: when a pipeline runs twice, it skips work that’s already done and can complete faster. This caching process happens automatically, sparing you the pain of maintaining fragile

X · Apr 6, 2026, 5:33 PM

[north dune]

Something mildly annoying: why does every failed run show graphql errors?

[sharp marsh]

graphql errors should only surface when something unexpected happen. Is there a chance you could share that trace URL with us privately? 🙏

[elfin frigate]

yeah it's up to the sdk to strip the extra wrapping away

[sharp marsh]

Seems to be a slip on the typescript SDK. I was able to come up with a simple repro 🙏

[winter fern]

has anyone tried to connect visual studio code to the dagger container to have a “remotely” option to explore/edit stuff on container from the IDE?

[sick monolith]

I think what you might be referring to is https://container-use.com/introduction @winter fern but i might be wrong

[winter fern]

not actually. anyway cu project seems its not maintained anymore so i would rather stick with dagger.

well my idea is kinda similar what cu offers and that is sandboxed env for coding agents - but i would rather here jus use dagger functions to get that - plus if possible to connect IDE to running dagger container to watch changes/ or modify directly some files, etc
kinda what is possible with dev containers on VSC

[sick monolith]

ahh i see

[cerulean token]

is there a reason why dagger develop with cli + engine versions v0.20.3 on a module using the Go SDK would produce a go.mod that depends on dagger.io/dagger v0.19.11?

is the idea that the Go SDK just hasn't changed since that version, so there's no need to bump it?

[sharp marsh]

Does the dagger.json have a v0.19.11 engine version by any chance?

[cerulean token]

no, that is also v0.20.3. dagger init --sdk go produces the same result as well

[sharp marsh]

yep just verified. I'd assume that was probably a slip. Doesn't seem to be the same in v 0.20.5. The SDK used is the same version

[tribal bison]

can i give a name to a withExec step? so that i dont get spammed with its logs but still can see them if i want to

[loud briar]

Is there a timeline on Modules V2? PRs look close/ready, I'm itching for these

[bold island]

Hello

I have a CI that embed 2 toolchains I have developed:

• protobuf
• golang

Since I migrated my CI to Dagger v0.20.4 the BuildImage function I have written does not work anymore because it does not find some sugar/entrypoints in the auto-generated code

• Action that fail: https://github.com/bcachet/zero_downtime/actions/runs/24342298949/job/71073985318?pr=2
  Here is the PR that brings the Dagger version bump that break the CI pipeline: https://github.com/bcachet/zero_downtime/pull/2

[bold island]

Can it be related to https://github.com/dagger/dagger/pull/11962 that has been introduced in https://github.com/dagger/dagger/releases/tag/v0.20.4 ?

[cunning jolt]

Thre is a 0.20.5, have you tried that?

[bold island]

Same issue with 0.20.5 as shown in this PR https://github.com/bcachet/zero_downtime/pull/1 with the associated Action failure https://github.com/bcachet/zero_downtime/actions/runs/24342286401/job/71073943942

[jagged flicker]

I've found the issue, but I'd say it shouldn't have been working previously 😕
Basically what you need is dependencies, not toolchains.
So just change toolchains by dependencies in the dagger.json and it's working

-  "toolchains": [
+  "dependencies": [

Dependencies are modules you want to use from the inside of a module, within the code. That's exactly what you have here.
Toolchains are other modules you want to install and use, but they are not exposed to the module's code. They are on the side of your main module.

[bold island]

ok, make sense

That was it, thanks a lot @jagged flicker

[cunning jolt]

I've noticed too that using a toolchain still does the codegen and can include it in code.

[steady sparrow]

Are there any docs on dagger generate (and the Changeset type) beyond this single paragraph? I'm failing to find them if they exist.
https://docs.dagger.io/core-concepts/functions/#generate-functions

[jagged flicker]

Not yet. Even if we are using it more and more, the command is still hidden by default and we are refining some of the bits (like automatically adding check function for every generate function)
But if you have any question feel free to ask and we can help

[steady sparrow]

Thanks for the offer of help! I think I've figured out my very basic usecase for the moment.

[steady sparrow]

For one, I'd like to be able to review the changes (clearly it knows, since the TUI is able to say how many lines are added/removed)

[jagged flicker]

Like to have a full diff of the changes before to apply them?
For that, currently, there's nothing builtin. But I'd love to have that too 🙂

[loud briar]

What's the difference between a function that returns a ChangeSet and a generate function (this use a pragma? +generate?)

[jagged flicker]

Yes, a "generate" function is a function returning a Changeset and flagged with +generate/@generate (depending on the language)
The main change is when you have multiple generate functions across multiple modules/toolchains. You can run all of them at once by running a single dagger generate.
All the resulting changesets will be merged together, it will fail on conflict, and then can be applied at once.
The primary goal of generate functions is to generate files that will be sourced-controlled (committed in the repository).
This is one aspect with check (and others that will come like ship) to allow to categorize functions for different use cases.

With that one very interesting advantage (same as check) is you just install a toolchain module that contains some generate functions, and they will just add themselves to the list of generate (that maybe you have defined). So at some point you only care about dagger check, dagger generate for most of the tasks and not dagger call and you don't have to write yourself a function that will aggregate multiple functions returning changesets.

[steady sparrow]

Random thought: not that this is a viable solution for all environments, but git diff accepts arbitrary files to present a diff.

[jagged flicker]

Yes, that could be a solution. From a Changeset we can already generate a patch file, so we should be able to load it on git and get the output. (changesets are using git under the hood, for instance to merge). We also have some utils to generate the summary.

[jagged flicker]

Not yet fully functional, user interactions needs some more work. But it's a start

[ancient ivy]

Hello everyone, I don't know if this is the correct channel, but if isn't I can remove the menssage.

I'm looking for a job remote or with visa sponsorship as Platform Engineer, I have more than five year as DevOps and used golang to write integration microservices, now I'm using dagger too, if you need in your team a Platform engineer to help, call me in DM or here that I send my CV

[winter linden]

I just created a #jobs channel so you can post there instead 😁

[ancient ivy]

Thank's

[granite latch]

Howdy, I'm just starting with dagger, migrating our CI to use this cool new tool. I've noticed that from time to time (when running buggy dagger code) that the dagger call command will crash really hard and take all of VScode with it (I'm running the command in the VScode terminal, btw). That is kinda bothersome, but I chalked it up to my buggy newbie dagger code. I switched to using the terminal proper (default terminal that ships with OSX) and the output is really glitchy; like, the screen jumps around to different scroll positions constantly. The arrow-key navigation works and when I go into a service's terminal everything is fine. Am I doing something wrong? Is there an environment variable or command line switch to make dagger chill out a little bit?

let's see if this posts.

that's what I'm getting in my eyeballs.

[cunning jolt]

Which version of dagger are you on?

[granite latch]

We are on 0.20.3

[cunning jolt]

I think the fix for that (for terminals without synchronized rendering support) went in 0.20.4 https://github.com/dagger/dagger/pull/12874. Can you try the latest version? 0.20.5?

Also I see that there is a revert for that same PR by @jagged flicker . https://github.com/eunomie/dagger/commit/d8fccf150871366f59450f5af1bbcb7729c37646. Maybe he can shed some light.

[granite latch]

will do, thank you!

(i'll start a thread in help next time)

[jagged flicker]

I haven't tried with the Terminal app, but yes maybe it was impacted.
But for sure the 12874 fixed multiple things, and there was also a follow-up to address some other issues.
The revert is something else, I have sometimes overlapping lines. Basically the lines that should be closed and disappear sometimes don't and the "result" of the function is written on top. But the revert was a wrong fix. And it looks like only a few are experiencing it, it might depend on the tools used (I'm tmux over ssh with ghostty)

[jagged flicker]

I never experienced it, but I know some people got issues, that have been resolved with the new tuist version.
If I'm right to use --progress=plain fixes it but that's just a workaround if you have to keep this version

[granite latch]

It looks like 0.20.5 has a more chill terminal. My eyes are happy tyvm

[granite latch]

dagger is really cool btw. I'm on the steep part of the learning curve still, but excited to level off.

[winter linden]

@granite latch thanks! We're actually flattening the learning curve aggressively 🙂

[torn wren]

Where is the best place to ask for help for and upgrade issue?

I have a module that works on v0.20.3 but on v0.20.5 I am getting a cannot import name 'Python' from 'dagger' error for context Python is a module that is developed by our team. It is installed as a toolchain in my local module (python sdk) and used as a type in my local python code? It does not seem to like from dagger import Python, but again it is all good on v0.20.3

[winter linden]

#1030538312508776540 is good, or if you prefer web, you can open a github discussion thread under "Help".

@torn wren I cross-posted for you: https://discord.com/channels/707636530424053791/1494081481671315496

[sharp marsh]

Any feedback on how we could make this better for future newcomers is super appreciated

[quick monolith]

Any suggestions? I needhelp troubleshooting my local dagger & ollama stack. The service launches correctly and responds with "ollama is running" when executing "curl http://192.168.0.236:8080" within the native host OS. It also successfully processes input from prompts after executing "ollam run llama3.1:8b". In a separate terminal . My Dagger shell also responds correctly when executing "llm|model" with providing the "llama3.1:8b ❯" input prompt, however when a query is made the error received is "Error: not retrying: Post "http://192.168.0.236:8080/v1/chat/compltetions": dial tcp 192.168.0.236:8080: connect: connection refused". I do notice that on the native host the inet interface for docker0 is inet 172.17.0.1/16 and I also have confirmed in another dagger shell being able to execute the command "container | from alpine | terminal" that its container interface is however inet 10.87.0.82/16. So I suspect a network issue in the container running the llm| model and "llama3.1:8b ❯" input prompt to receive connection refused error? Any pointers would be appreciated .

[granite latch]

One thing that struck me on day one was the lack of a "full owl" example implementation. The docs have several trivial examples of single service DAGs, meanwhile my stack is composed of like 12 services, several of which fit the trivial database + app pattern, but the real ones are not represented in the docs. It would be a welcome addition if there were (easily found) full examples of realistic working stacks which illustrate the downright fucky patterns that you'll find at startups and enterprises everywhere.

[winter linden]

Yes agreed... Our issue is that most of the realistic ones are closed source... But we are working on making our own CI a "golden example". We do use Dagger heavily to build, test and ship Dagger... But we are an extreme use case, because of the nesting (Dagger in Dagger... sometimes 3 layers deep). So in that way not the most representative. But in other ways (the complexity... the bagage... the pragmatic tradeoffs...) we are 🙂

[winter linden]

@here ⚠️ MERGED! Dagger has officially removed Buildkit ⚠️

We have merged "project theseus", aka the removal of Buildkit from Dagger. This is a multi-year effort that is finally coming to an end.

Now we are starting a baking period before the next stable release. This is a big enough change that we will need more real-world testing before release. If you are interested in early testing from main, please me know. We are going to improve the pre-release system so that it's easier to use a dev version of dagger.

Thank you all! Lots of exciting features just became much easier to ship.

https://dagger.io/changelog/#project-theseus-removing-buildkit

[fossil pine]

I could integrate it into my daggerverse nightly test

[floral pike]

Is the name 'Project Theseus' a reference to the Ship of Theseus paradox, replacing every plank of the ship one by one until nothing of the original remains ? 😜

[winter linden]

https://giphy.com/gifs/star-trek-leonard-nimoy-mr-spock-sR74mVSQe1ems

Yes exactly 🙂 We've been removing buildkit piece by piece for over a year

[loud briar]

Get me an engine/cli built from main and I'll put it through it's paces on our workloads (docker building, microservice deployments, java testing with postgres dagger service) Monday morning

If you get chance to merge modules v2 before then even better 😉

[winter linden]

@loud briar yeah we're going to enable pre-release testing on 2 tracks:

• On main (to test theseus and anything else we merge on top)
• On the modules-v2 branch (special case for testing before merging)

That said the first chunk of modules-v2 should merge soon, now that theseus is out of the way (we had to freeze main for a little while)

[loud briar]

Sounds good to me. Is Theseus a case of "you shouldn't notice anything at all" when building images with DockerBuild?

[winter linden]

yes the goal of theseus is to be 100% backwards-compatible. Hopefully the only thing you'll notice is that it's faster and more reliable 🙂

dockerBuild in particular is an area that needs thorough real-world testing: using buildkit gave us Dockerfile compat for free.. So we had to rebuild a buildkit/llb compat layer over the dagger API 😅

The good news is that now dockerBuild will be less of a black box: full tracing, better integration with dagger caching etc

once we roll out scale-out for example, it should be able to scale out a single docker build across multiple machines on the fly 🙂 (if workload profile justifies it)

[loud briar]

Better caching could do a lot of work. We've got a range of images being built in Dagger, I'll test some and report back

[cunning jolt]

I'm keen to test all of this too. I'm in the same boat as MB, I can test better if there's a CLI and image version I can use ootb instead of building it

[winter linden]

@loud briar @cunning jolt this is the plan: https://github.com/dagger/dagger/issues/12996

[loud briar]

I did see that, wasn't sure where it'd be prioritised / how much work to get it released

[cunning jolt]

And if it'll actually work in my network 😅

[winter linden]

Ha, well.. that I don't know 🙂 It's just a regular http download of a release tarball. From your regular host context (CLI-side)

[cunning jolt]

It's the engine that I'm not sure will work. I have my custom engine (CA Certs). May be able to get away with some simple testing

[winter linden]

Ah I see. Yeah it will rely on the CLI's standard engine bootstrap

A good reminder that we really should make it so you don't need a custom build of the engine for environmental reasons

[cunning jolt]

I'd love to not have to maintain my own.

[winter linden]

Is it only for the custom certs? What are the blockers to using an official build?

[cunning jolt]

I think it's just custom certs. We used to need the proxy configs but now you can define them in the _EXPERIMENTAL_DAGGER_RUNNER_HOST.

[tired moth]

Is this released? Please share @winter linden

[tired moth]

dang repo shoul be in dagger org in github for more reliability and better official recognition. Just a suggestions. Thanks

[tired moth]

DAGGER ROCKS, SOLOMON ROCKS @winter linden

[white zephyr]

I am interested in testing, github account is awdemos.

[royal flume]

Is the dagger mcp feature broken in 0.20.6. Earlier the custom functions would be available through mcp. Now this is not the case. Only dagger core is available and list tool comes back empty.

[cerulean token]

Has anybody created a Dagger module for kind?

EDIT: I see the k3s one from github.com/dagger/dagger/toolchains/helm-dev/k3s, but I'm having some trouble with it:

❯ dagger -m github.com/dagger/dagger/toolchains/helm-dev/k3s@v0.20.6 call --name "example" server up
▶ ✔ connect 0.2s
▶ ✔ load workspace: . 0.9s
● ✔ parsing command line arguments 0.0s

● ✔ with(name: "example"): Query! 0.0s

▶ ✔ k3S(name: "example", image: "rancher/k3s:latest", keepState: false, enableTraefik: false): K3S! 1.0s
▶ ✔ .server: Service! 1.2s
▼ ⣽ .up: Void 8.4s
┃ 00:18:03 INF tunnel started port=6443 protocol=tcp http_url=http://localhost:6443 description="tunnel 0.0.0.0:6443 -> a7coa2tc8df9o.jgk1tan52vivc.d
┃ ger.local:6443"

And then, from a separate terminal:

❯ curl -v http://localhost:6443
* Host localhost:6443 was resolved.
* IPv6: ::1
* IPv4: 127.0.0.1
*   Trying [::1]:6443...
* Connected to localhost (::1) port 6443
> GET / HTTP/1.1
> Host: localhost:6443
> User-Agent: curl/8.5.0
> Accept: */*
>
* Empty reply from server
* Closing connection
curl: (52) Empty reply from server

[slender star]

I just tested dagger part via

curl -fsSL https://dl.dagger.io/dagger/install.sh | DAGGER_COMMIT=e7d80929a47807a320b3a33add4733a59d4d9d3e BIN_DIR=/tmp/dagger-dev sh
# now dagger is in /tmp/dagger-dev
git clone https://github.com/kpenfound/demo-react-app
cd demo-react-app
/tmp/dagger-dev/dagger check
# run again
# change index.html a little
# run again

Looking good! 😍 Congrats, team! 🎉

[winter linden]

Thanks Jeremy!

The upcoming workspace branch has improvements on the toolchain system in part inspired by your feedback to my last demo fyi

[white zephyr]

"when does dagger build itself with dagger?" sorry I'm in a recursive mood, ignore me.

[winter linden]

Dagger is indeed built and tested with Dagger 🙂

By "when" do you mean "at which point in our CI pipeline"?

[white zephyr]

purely tongue in cheek, I've been watching a how the rust compiler installs itself in rust and it got me thinking about slef-compiling software, purely offotpic.

[loud briar]

Will ModulesV2 workspace config files have the system env var fallback that env files have currently?

[winter linden]

I would say yes since it seemed quite useful in .env. Is that your preference?

[loud briar]

Yep, works a treat in Gitlab. We can set variables in jobs/templates/components and workspace config can pick them up seamlessly

[winter linden]

Do you have .env checked in as-is, or do you generate or rename the file dynamically in the CI config prior to calling dagger?

[loud briar]

Depends on module, but as workspace config becomes available it'll all be checked in

[winter linden]

FYI workspace config already has a built-on concept of multiple environments. So you can do:

[modules.foo]
settings.toolVersion = "42.42"

[modules.bar]
settings.extraPackages = ["a", "b"]

[env.gha]
modules.foo.settings.token = "op://..."
modules.bar.settings.extraPackages = ["a", "b", "c"]

dagger --env=gha check

[loud briar]

Yeah will be relying on that heavily because we authenticate differently locally vs Gitlab

[winter linden]

same 🙂

[royal flume]

Anyone ?

[winter linden]

Sorry for missing this Rishi. It's possible that there's been a regression. Would you mind filing an issue please 🙏

[winter linden]

@royal flume I was able to reproduce. Filing the issue. Sorry about this

[loud briar]

How far off is this? I notice some movement from AlexCB on merging workspaces, is that this?

I'm basically planning to install Dagger from main once it's merged and get converting our stuff

[winter linden]

Yes, that's what AlexCB and Guillaume are working on. Tibor is wrapping up release engineering work for 0.20.6 (which was its own can of worms), then joining the effort. We're in the last stretch. It's a big diff so we're being careful. Everything in #1494504722898354326 is related to this

[winter linden]

We're being extra cautious because it's several changes in one:

• Changes in repo layout, which will require migration (very rare for us, the config file format has been pretty stable)

• Changes in UI

• New docs (current format and flow doesn't make sense with the new capabilities - in a good way, easier to get started without writing custom modules etc)

• Big diff internally (although we got ahead of it by merging workspace-api and workspace-plumbing earlier)

• Now rebase on top of theseus (buildkit removal) which adds its own churn

Choppy waters because of all the changes, but it will be worth it 🙂

[royal flume]

Thanks for the quick fix. I was going to post it on gh but wanted to confirm it first.

[winter linden]

Not a fix yet 🙂 But the first step towards one 🙏

[sudden ocean]

I have a dagger dependencies like

main module
  | toolchains
     | app 1 
        | toolchains
           | share module

I annotate check at the share module function. When I run check -l at the main module, I didn't see any check from app 1 but see the check when change working directory to app 1. Is that a limitation of check (or toolchain, i don't know)?

[jagged flicker]

I think that's expected. On main module you see the checks provided by your main module and the toolchains modules, so app 1 module here. But if the check is in share module it's not in the app 1 module and they are only visible from the app 1 module itself.
(I hope that's clear enough 😅 )
Basically if you want checks from app 1 to be exposed to main module, they need to be exposed by app 1 directly, so inside app 1 module code. Toolchains of toolchains are not visible from the main module.

[sudden ocean]

@jagged flicker Thank you for explanation. It is very clear. 🙏

[jagged flicker]

The new modules v2 / workspace will help to understand this, by making visible the notion of workspace. It will help to understand the scope of a module, where it can be used

[winter linden]

Yeah this scenario is one of the reasons we're removing the distinction between "regular modules" and "toolchain modules", and just simplifying it under "modules"

[winter linden]

@royal flume fix is up: https://github.com/dagger/dagger/pull/13030

[tired moth]

Dagger watch when?

[plain barn]

I noticed Dagger doesn't allow a module to return types from other modules - what's the reasoning behind this?

[winter linden]

I think we will eventually (perhaps soon) but there is a tradeoff related to dependency hell... See https://github.com/dagger/dagger/discussions/12159

[loud briar]

We work around this currently by storing the parameters needed to create the (submodule)type on the module struct so it can be constructed anywhere in the module, rather than constructing it once and passing that type around. Would definitely be a nice-to-have but it's not my highest priority

[plain barn]

I see, thanks, that was my guess as well.

Yeah it definitely hurts modules usability for my use case.

[winter linden]

The general trend is towards modules that are less project-specific, handle more dynamism and composition themselves, and are more often directly user-facing. So, more like "plugins" than libraries (although the library use case will continue to exist - we just think it will be less load-bearing than today).

In this usage pattern, I've found the lack of this feature to be less of a problem. But, it would still be nice to add it 🙂

[fossil pine]

Tests are (mostly) passing with the latest main build of Dagger: https://github.com/sagikazarmark/daggerverse/actions/runs/25101149813

(I assume it's already the buildkitless version)

[winter linden]

SHIPPED: faster engine connect. If you're using the docker driver, sessions connect twice as fast now. https://dagger.io/changelog/#faster-dagger-connect

COMING SOON: a lockfile for your CI 😇 . https://dagger.io/changelog/#a-lockfile-for-your-ci

[cunning jolt]

The PR/commit linked to this seems to be reverted - https://github.com/dagger/dagger/commit/e73eb490c63a442cef06d18afeae69dc0782688e. Did it go in as part of another PR?

[cursive bridge]

Did something change with 0.20.6 and constructor args? I have args on my test constructors which do not seem to be recognized anymore. I'm following standard all entrypoint for all tests, and i have optional args on the constructor (New). The args are no longer recognized after upgrading to 0.20.6, and trying to do a dagger call all --help provides help text, but doesn't list any of the constructor args.

[cursive bridge]

I downgraded back to 0.20.5 and ^^ was resolved.

[elfin frigate]

what about dagger call --help? constructor args are normally passed to call itself, we did a large refactor prepping for modules v2 so that code definitely changed. maybe you were passing constructor flags to all instead of call before and that used to work?

[cursive bridge]

The line that used to work but failed in 0.20.6 was dagger call <args> all

I do see the args in dagger call --help, but they weren't recognized in my pipelines. Now, however, when I try locally they are. I'm very confused. I can try upgrading in cluster again and see if I can repro.

[winter linden]

Well that's embarrassing... Yes it looks like we had to revert it because it broke podman users? I will change the entry to "in development". Sorry and thanks for catching!

0.20.6 & constructor args

[cunning jolt]

It's all good! Tons of moving parts, so I understand.

[fossil pine]

Can a toolchain interact with anything other than the workspace/filesystem?

For example: can I pass a container built in my module to a toolchain function (check) (eg. using customizations)?

Use case: install security scanner toolchain and scan container images built by modules automatically.

[winter linden]

Not yet but we have several use cases that call for it:

1. Configure playwright module to target a dev service
2. Configure GraphQL API doc generator module to target our dev engine
3. Scan container images automatically (your use case)

Question @fossil pine . In your use case do you picture a discovery mechanism + unconditional scanning of whatever is discovered - or, an explicit selection of certain artifacts to scan. I ask because those are actually 2 different design problems: the discovery; and the selector syntax.

[fossil pine]

Good question. I'm gonna say both. but if I have to pick, I'm gonna say discovery.

I have a few different use cases in mind, scanning is one

probably the simplest one TBH

but the "real" use case would be "gather whatever artifacts this module produces and publish them/run a specific workflow"

hopefully I can catch @sharp marsh next week as well and I can show him a few things I have in mind

[winter linden]

In the case where you really want to publish all the artifacts unconditionally, probably we're not talking about a "ship" action (production deployment or release), because that would typically require some project-specific selection ("publish this artifact to that target). Right?

[fossil pine]

correct

I thought this is what customizations are for, combined with workspaces

[winter linden]

In a general sense yes, absolutely. But we still need to design the right pattern for each use case.

And specifically: along what lines do we break out reusable modules? There are many possible separation lines in a software stack, with lots of overlap

@fossil pine if you haven't read it yet -> this is where the final design puzzle is being assembled for that https://github.com/dagger/dagger/pull/12900

[fossil pine]

Thanks, I'll give it a read.

Q: the workspace plumbing is still in progress, right? No config.toml file yet? What do we have instead.

I'm working on a Rust workspace and I thought I'd give dagger workspaces a try

[winter linden]

You can try a build of the workspace branch:

dagger call -m github.com/dagger/dagger@workspace engine-dev playground terminal

Then inside the playground, try:

• Git clone of your existing daggerized repo
• dagger -W URL_OF_REPO (remote workspace selection)

We're also going to add a flag to the dagger CLI so that you can seamlessly switch to a dev release

[fossil pine]

Are dev releases published somewhere as well? If so, they could be added to Nix. 😃

[fossil pine]

@elfin frigate sorry, in the middle of something, can't open a proper issue now, but I wanted to leave a note:

This complains about the nullable value:

    if (self.source != null) {
      container = container
        .withWorkdir("/workspace")
        .withDirectory(".", self.source)
    }

This works:

    let source = self.source
    if (source != null) {
      container = container
        .withWorkdir("/workspace")
        .withDirectory(".", source)
    }

Not sure if it's intentional, but it seemed a bit weird to me

[elfin frigate]

known limitation, yeah - the null flow typing only works for simple variables atm, not arbitrary expressions

[elfin frigate]

actually there's an issue with supporting that specific example: self.source could be non-null on the first call, but null on the second call

seems unlikely in practice esp in the context of Dagger, but definitely a wrinkle in terms of language semantics. i'll think about it, i'm not a big fan of the complexity and limitations with null flow typing, maybe i can find something better

[sharp marsh]

not for the moment. We have a meeting this morning to setup this dev / nighly releases flow

[winter linden]

@fossil pine it's a continuous build straight from main. Going through a full release process would be too much friction at that pace (literally every time we merge to main)

[fossil pine]

I wonder if using .env for locking tool versions is a good idea: function accepts a version argument, .env locks it.

THe downside is .env is supposed to be user default, right? So if I commit it, it becomes a project default, users need to get around changes to .env (ie. not commit it accidentally).

Or should I just use customizations (for toolchains)?

[winter linden]

do you mean locking down to the digest? or version number?

for version number: IMO a module setting (constructor argument) works great. Today you can set that in dagger.json customization. With workspaces it becomes a super easy toml config field

[fossil pine]

well, version number at least. what if I use a bunch of tools for building my app? preferably I want them to be locked down, but overridable (but maybe I'm overthinking it)

so dagger call my-linter runs the locked version

dagger check my-linter --version whatever runs something else

[winter linden]

yes agreed. It's just more of a configuration file than a "lock file" use case

in some cases you might want a version matrix? Was considering that for the new github.com/dagger/go module

[fossil pine]

oh yeah, there is that as well

BTW I'm building abominations today 😄

I just wrote a dagger module, that runs a nix container, installs the docker cli and builds containers, talking to the docker socket on the host 😄

don't ask 😄

[jagged vault]

Hi, I just upgraded to latest v0.20.7 engine, but having SDK codgen issue on project update !
Am I missing something ? 🤔
Is it related to the "proxy" module dependency ?

{
  "name": "hello-dagger",
  "engineVersion": "v0.20.6",
  "sdk": {
    "source": "typescript"
  },
  "dependencies": [
    {
      "name": "cypress",
      "source": "github.com/quartz-technology/daggerverse/cypress@d58a027ddcaf210b6e2c5feba307c3b48fdb0255",
      "pin": "d58a027ddcaf210b6e2c5feba307c3b48fdb0255"
    },
    {
      "name": "proxy",
      "source": "github.com/kpenfound/dagger-modules/proxy@proxy/v0.2.6",
      "pin": "428ffe16f0a2e3c9e4d0d9debf6420db29d3cb64"
    }
  ],
  "source": ".dagger",
  "disableDefaultFunctionCaching": true
}

Thanks.

[sharp marsh]

cc @jagged flicker @wispy tapir

[jagged vault]

Okey, just tried to remove the proxy dependency and the update went well this time.

Certainly a module version schema mismatch (reinstalling it triggers the same error...) ?

[sharp marsh]

checking what's up with that module @jagged vault

[sharp marsh]

@jagged vault yes, that module is way too old. @warm temple any chance we could bump the Dagger version there?

ref: https://github.com/dagger/dagger/blob/main/engine/version.go#L33

and the module is on v0.18.x

[warm temple]

bumped!

[jagged vault]

Yep, it's good with v0.2.7 ! Thanks 👍

[jagged vault]

Some general questions :

1- What is the .sync() exactly for : "forcing the execute to ignore default lazyiness" => in simpler words ?
2- Why do we need a final run in command .withExec(['npm', 'run', 'test:unit', 'run']) ?
3- the package.json file for init TS SDK, references "typescript": "5.9.3" dependency from where, global node modules cache on host ?
4- where are EngineCache Metadata located on host ?
5- the Data Layer is related to engine API calls being cached ?
6- What is the Dagger Core API (apart from Engine and Modules) and when does it intervene ?
7- If I'm right, the Live Development is the works ?
https://github.com/dagger/dagger/discussions/12400
#1136087508732616764 message

[tender timber]

Hey guys, anyone mind helping me ? https://discord.com/channels/707636530424053791/1495875753093501018

[torn wren]

I have a breaking change when upgrading from v0.20.3 → v0.20.7 and wonder if there is any option that does not require my users to make a coding change? We have the dagger enging running on a Gitlab runner and if I upgade to v0.20.7 it will break all the build until users make coding changes.

Before on dagger v0.20.3 the code below was valid.

import dagger
from dagger import dag, function, object_type, check, DefaultPath, Doc, Ignore
from typing import Annotated

SrcDir = Annotated[
    dagger.Directory,
    DefaultPath("/"),
    Doc("The source directory of the project"),
    Ignore([".venv", "**/__pycache__", ".dagger"]),
]

@object_type
class Dagupgrade:

    @check
    @function
    def myfn(self, src: SrcDir) -> dagger.Container:
        """list src directory"""
        return (
            dag.container()
                .from_("alpine:latest")
                .with_directory("/src", src)
                .with_exec(["ls", "/src"])
        )

Now on v0.20.7 it seems that dagger is more strict with types. Using SrcDir Annotated as a global is no longer valid. This is done for code reuse to make code DRY (many functions need the same SrcDir argument). It gives the following error:

$ dagger check 
✔ load module: dagupgrade 0.2s
Error: failed to get schema: failed to load schema: failed to get schema for module "dagupgrade": failed to create function "myfn": find mod type for function "myfn" arg "src" type: "DagupgradeSrcDir!"

If I in-line Annotated in the function signature it works but then I would have to copy that in many places. I have two question:

Q1: Is there a better workaround that keeps the code DRY
Q2: Is there is any way to allow the old code to work on v0.20.7 perhaps with a env var flag set? This would let me upgrade the engine and give users time to make the coding changes.

[jagged flicker]

We are currently working on fixing it. v0.20.7 comes with a new AST analyzer that looks like to have some gaps.
This specific issue should be fixed by PR #13093
An issue #13097 is tracking the work on it.
The final fix should be in PR #13095

The goal is that no code change should be necessary, the issue is on the new parser not on the actual code

[winter linden]

Sorry about that @torn wren ! The upside is that the Python SDK is on track to be more efficient and much faster 🙂

[torn wren]

Nice! I did not expect the "no code change" answer! i will follow the PRs, thanks

[loud briar]

As an aside @torn wren, the new workspace arg should prevent the need for that global SrcDir and you can just global the exclude list?

[loud briar]

CacheVolume questions... CacheVolumes only match on cache key for that volume, nothing else? Function inputs differ + cache volume key identical = cache volume is attached to the container using WithMountedCache? Cache volume size unlimited until engine instance starts garbage collecting?

[winter linden]

Yeah you can change it to this @torn wren :

  import dagger
  from dagger import dag, function, object_type, check


  @object_type
  class Dagupgrade:

      @check
      @function
      def myfn(self, ws: dagger.Workspace) -> dagger.Container:
          """list src directory"""
          src = ws.directory(".", exclude=[".venv", "**/__pycache__", ".dagger"])
          return (
              dag.container()
                  .from_("alpine:latest")
                  .with_directory("/src", src)
                  .with_exec(["ls", "/src"])
          )

ws: dagger.Workspace is treated as a special case, the engine will inject the current workspace into the argument, allowing you to dynamically access the workspace filesystem without anything getting uploaded before hand

[fossil pine]

@elfin frigate Do you think this is possible? https://github.com/dagger/dagger/issues/13102

[obsidian lake]

Hey guys, ran into the runc bug today. I also noticed a lot of zombie git process, but I'm not sure if these are of concern - thought I'd let you know.

[elfin frigate]

Dang: adding with?

[dusty pelican]

I think I know the answer to this but want to get some feedback from the pros before trying to run through a PoC on adopting dagger.

We have built out our CI/CD purely on GitHub Actions. Atomic Functions / Actions are JS Based and then strung together in reusable workflows. TypeScript SDK probably will be the easiest transition from our JS Actions?

Any pros/cons between the 3 "Official" SDKs?

[coarse hatch]

v0.20.8 is out, it reverts the python sdk regression in v0.20.7.

[torn wren]

good news, i am traveling but will check it out asap

[plain barn]

Are there any plans to provide an official LLM skill for Dagger? I noticed you had an internal one (I think) for Dagger development

[tired moth]

as dagger team is replaced buildkit so do we need separate container runtimes or not?
and new dagger's native container runtime will be inbuilt soon?

[winter linden]

Yes we are actively working on that - since the UX is in the process of changing with modules v2, we're working out what version of the UX to aim for

[winter linden]

Buildkit replacement will cause on architecture change.

• Before: engine is a single packaged OCI image, with private bundled buildkit
• After: engine is a single packaged OCI image, with no bundled buildkit

[tired moth]

so all container runtime requirement will be there like docker, apple container etc, that is not affected, right?

[winter linden]

Correct, no changes to container runtime requirements.

[cursive bridge]

I notice that the dagger halm chart hard codes the replicas to 1 for the statefulset option. From the comment, that is largely a concern if the user is using hostpath for storage but not a problem if using pvcs. Is there an objection to allowing the ss replicas to be set by a values.yaml field and the chart defaults to 1? The comment should stay in the ss definition, but seems like it would be good to allow multiple pvc ss in the chart.

[sharp marsh]

seems reasonable. Is this how other stateful services approach this? Happy to review whatever is generally considered a good practices here. I'd check what other services like redis, memcahced, etc which don't have custom operators are currently doing for sts

[blissful stratus]

Are there any recent plans to providing options to limit the resources of services or execs or somehow controlling the parallelism of certain expensive executions?
We are having trouble constraining big monorepo-builds that contain kind of expensive integration tests (using services or dind testcontainers) on multiple modules.

[lucid ruin]

Has anyone else here tried running bun as the main container for dagger functions. I am having a strange issue where I cant do a bun install in a dagger function because the bun install process finishes but then bun goes to sleep and dagger never manages to continue on after that.

edit: Figured it out. Bun was failing to resolve private packages. And there is a bug that is fixed in 1.3.14 that will just quietly wait forever after the install fails on those private packages. Only reason this was hapening in dagger was because the auth for those private packages was mssing in dagger.

[tired moth]

Hi @sharp marsh and @winter linden I’m reaching out to request a final look at PR #13133, which adds support for the Incus container runtime.
As Incus gains significant traction for system containers across Linux distributions, this integration allows Dagger to reach a broader audience of users who require robust, distribution-agnostic environments. We’ve ensured the implementation follows existing patterns to keep the maintenance footprint minimal.
Is there anything further needed from my side to get this over the finish line? We’re excited to see this land and help expand Dagger's runtime flexibility.

https://github.com/dagger/dagger/pull/13133
https://linuxcontainers.org/incus/docs/main/container-environment/

[sharp marsh]

give us a bit of time to check this one @tired moth. It's already in the review backlog and we'll get to it as maintainers have bandwidth

[tired moth]

Waiting for that time only.
Thanks for considering.

[tired moth]

Dagger is going to introduce dagger's inbuilt container runtime of its own soon?

[torn wren]

I upgrading from v0.20.3 to v0.20.8 and I am seeing my cache fill up. On v0.20.3 it would clear itself out after reaching a certain disk usage threshold but now on v0.20.8 it seems the cleanup is not happening. For reference here is our engine.toml

[grpc]
      address = [ "tcp://0.0.0.0:9090" ]

[log]
      format = "json"

[worker.oci]
  all = true
  maxUsedSpace = "130GB"
  reservedSpace = "10GB"
  minFreeSpace = "50%"

is there any change in > v0.20.3 that we would think related to this issue?

[swift inlet]

Seems related / similar to #1495875753093501018 message, I've been having a hard time reproducing it. Can we please keep the conversation on this thread 🙏 There's also, from my last understanding, 95% chance this is solved by theseus (next release)

[exotic hearth]

Has anyone used dagger cloud's observability to eliminate flaky tests? I absolutely hate the fact that these exist and are just accepted as "run it again"

[winter linden]

Yes and agreed 🙂 We have several test-related features in the pipe 🙂

In fact @exotic hearth just last week we shipped a new "Tests" view in individual traces. If your test tool emits standard OTEL spans for test instrumentation (there is a new OTEL spec for that), then Dagger Cloud will render the individual tests and test suite in the trace

--> We are developing Dagger modules that you can drop into your project, and automatically instrument your native test runner so that it emits the necessary otel spans, with zero code change required

This in itself does not solve flaky test management, but it gives us a solid foundation to build features towards that end

TLDR tests are becoming a first class citizen in Dagger

[exotic hearth]

this is really cool, I was thinking about doing something similar with ebpf and self hosted runners for observability/debugging but this is way beyond that it seems

[winter linden]

@exotic hearth to see an example, you can look at any of the traces from our own CI (they are public)

Random example: https://dagger.cloud/dagger/traces/6fb733ae9c9d3825d129ca2d307282e3?listen=b24c21fcdf7a2b95&suite=github.com/dagger/dagger/core/integration&viewMode=tests

Note: passing tests are collapsed by default but you can expand

[exotic hearth]

This is amazing, testing and debugging ci has been such a pain in the past

[winter linden]

The nice thing is that it's all one trace so you have the full context:

• If you only care about the tests, you can focus on that
• If you care about the test environment -> how it was built, what its dependencies were, why it was slow, you can look at the surrounding system trace

btw these traces are also emitted by local runs

And, they support dagger-in-dagger nesting.

So for example, if one of your tests needs to build and test a dependency on the fly (for example our own tests need to build and run an ephemeral dagger engine 🙂 ) -> the dagger calls for that will show up in the context of the test , and you can keep driling down in that trace, as deep as necessary

I think @elfin frigate is working on upgrading the TUI also with test view (all the data you see in the TUI is rendered straight from the same OTEL stream)

[exotic hearth]

Awesome, really excited, I'm going to start labbing this and testing it out

[winter linden]

@exotic hearth what's your test runner of choice? I can tell you if we have a module already available for it, or in development

[exotic hearth]

We were just using differnet ones at my last position, some self hosted in an arc controller with custom images, and some github hosted runners, not sure what the new position is going to be

someone mentioned there's some circleci laying about, and especiallyif people aren't set in stone on which platform they're using, something agnostic is going to be a huge help

but I've gone through the nightmare of trying to manage hundreds of repos with reusable workflows and composite actions with github actions and when this new place wanted me to take over CI/CD I started researching because i don't want to go through that mess again, and dagger seems like a great thing to test

[winter linden]

@exotic hearth I mean test frameworks... like go test, pytest, jest, vitest etc

[exotic hearth]

oh sorry, go test

but actually this new place is going to be RoR so who knows 😮

[winter linden]

Then try this module: github.com/dagger/go

[exotic hearth]

but I prefer go yeah. thanks

[plain barn]

any chance secrets manager calls could be batched across functions (checks)?

I need to authorize 1Password for each individual Dagger check when I run dagger check because the module takes a secret (as a module-level field). I have around 5-8 checks depending on the repo now so it's quite annoying.

[warm temple]

I'm 100% with you. It actually worked like that initially (only one authorization) but it was too slow because it would then request each secret sequentially, and the change to request them all at once meant the client was never pre-authorized. I'll have a look and see if there's a better batching option with the client these days

[winter linden]

MERGED we have a lot of fun improvements piling up for the next release 🙂

• A lockfile for your CI. Every time your pipeline looks up a container or git tag, that look up gets persisted in a dagger-specific lock file. So you can enforce immutable tags at the CI level, across the whole pipeline

• REMOVING BUILDKIT. Dagger is replacing BuildKit's solver with its own native engine, unlocking robust remote caching, cache observability, and more flexible module execution.

• If you useddagger check and dagger generate: now they play well together. By default check will fail if you have stale generated files.

• Speed improvements. Removing buildkit makes a huge difference... And we merged other optimizations as well 🙂

• Lots of polish and reliability fixes.

CHANGELOG: https://dagger.io/changelog/#next

We also shipped many improvements to DAGGER CLOUD this month:

• Test-aware CI! Wouldn't it be great if your CI understood your tests, and let you visualize and analyze them? Well Dagger Cloud now does that 🙂 https://dagger.io/changelog/#cloud-tests-ui

• Every trace page has an easy to copy dagger trace command, to visualize it in the CLI https://dagger.io/changelog/#cloud-trace

• Cloud Engines (in early access) has a v2 scheduler. More efficient compute and cache allocation, and a contininuous tuning system: the more you run, the more your scheduler adapts to your workloads . https://dagger.io/changelog/#cloud-scheduler-v2

• Cloud Checks (our native CI bypass) now uses a single unified pool of engines. More efficient and faster. https://dagger.io/changelog/#cloud-unified-pool

• We fixed glitches with check re-runs and engine version display https://dagger.io/changelog/#cloud-2026-05-stability

If you want magical auto-scaling Dagger engines in the cloud, DM me for early access!

https://dagger.io/changelog/#cloud-2026-may

[tired moth]

This message is just a gentle reminder. Thanks

[arctic fox]

Any chance there's a Go package for parsing the --progress plain format? Or is there a parseable format planned in the next release? Agents are able to analyze the plain format but want a tool for querying on the trace of a finished run so I can manually validate agent analysis or trim the data down which my agents need to analysis to help save on tokens. My use case is having agents help me better optimize the setup and execution of my personal (and at work) daggerverse tests in a resource constrained environment (i.e. Github actions and Jenkins)

[winter linden]

Good question, we've got several features in the works for making Dagger more directly useful to the user's agents.

There is a trove of telemetry data in your dagger cloud account (if you enable it) and we've only scratched the surface on letting you (and your agents) consume it for maximum insights

[arctic fox]

Also in case it wasn't implied dagger trace is an amazing command

[winter linden]

ha ha I agree 🙂 one of the pillars we're building on

@arctic fox if you liked dagger trace you should check out https://github.com/dagger/dagger/pull/13130

[arctic fox]

I was trying to guess some hidden subcommands today 🤣 but more seriously anything available in the interim?

[winter linden]

If the goal is feeding telemetry data to your agent, best is to experiment with different progress renderers:

--progress=logs
--progress=report

are most llm-friendly

all output formats are rendered from the same source of truth: otel telemetry stream.

If you want, you can also tap directly into that either standard otel tooling

(be warned that raw otel ecosystem is not for the faint of heart)

[arctic fox]

... is report in the list presented by cli?

[arctic fox]

I'm one of the core authors of a framework at work and contribed to opentracing pre otel. I also demand otel support so well aware lol. The go otel sdk is insanely easy to work with though. The biggest hurdle I've seen is out of central analytics aka local analytics.

Every vendor provides everything but cuz otel doesn't have a std "stdout" format its hard to get off the ground unless youre already familiar with the ecosystem

[loud briar]

Any timelines on workspace config coming up this week? Scope of that feature seems to have grown a few new limbs

[winter linden]

in our case you would just set the standard env vars and dagger CLI will honor them and send the raw stream to your collector. I think there's a way to get it via the daggwr sdk too?

we send spans, logs, and metrics

[arctic fox]

Oh yeh I know dagger works with std otlp. We use it with our jenkins env. Im more interested with tooling around the post-ci side at the moment. Hence the parse progress plain parser question

I think a json or other structured output would be ideal. I think sacrificing real time updates is worth getting structured output... obviously just my opinion though... might help limit logs in ci systems that have limits though e.g. gitlab ci

Is there any plan to offer dagger cloud for self hosting?

[winter linden]

@arctic fox do you use dagger check ? those checks are fully introspectable in the API

[arctic fox]

Yes but my use case is extracting span+logs within a specific trace. And doing so in a minimal way. Not sure if im missing something with the check command but so far I haven't been able to replicate what im looking for. Ive also been looking for improved caching which I think the next release will take care of

So far ive replicated go test behavior within my own daggerverse i.e. all modules are concurrent but each module is sequential. Its the per module tests that im trying to optimize. For some reason the ports are taking some time to become healthy. Note this is dealing with kafka so the port is the exact same in each test. My suspicion is that is the root cause

[arctic fox]

My next plan on top of creating a parsing library for --pogress plain was to create a skill for interacting with the api

Does dagger by chance already have an MCP?

[arctic fox]

This is the specific use case where I'm wanting span specific filtering on top of dagger trace since the kafka spans are the ones where port "conflicts" are occurring. This is where my knowledge is more theoretical since I'm not 100% sure how networking works within the engine but based on observations it appears every container shares the same interface which means if 5 containers need the same port then they will all wait until that port is available. Again I'm not 100% confident of this stance. Just basing it on my personal CI/local experiences

[cunning jolt]

Hey @winter linden , @arctic fox is a colleague of mine at Fidelity 🙂 His teams are starting to pick up dagger.

[arctic fox]

@winter linden im also interested in contributing private maven registry support cuz most of my team is Java. I can help hash out a design then implement it if one hasnt been created yet

This would apply more broadly for @cunning jolt and myself (aka 100+ teams)

[tired moth]

@winter linden @sharp marsh i request first to review and merge already contributed pr rather then focusing on new planning. This message is just a gentle reminder only. Thanks

[sharp marsh]

@tired moth we generally prioritize our backlog based on user requests and demand. We'll eventually get to your contribution but please don't push us on getting it merged as the team is already spread too thin improving other parts of the platform.

As any other OSS projects, if you really need this, you can run a fork with your change and rebase with upstream changes as needed. Given that this contribution is well scoped and not that large, it should be straightforward to do that

[tired moth]

alright thanks

for any project/repo, for all cicd code written with dagger should be written and maintained in .dagger folder only? is this the new way and standard we are going to follow with dagger?

[tired moth]

@sharp marsh is this correct?

[sharp marsh]

not fully correct. This whole UX is about to change very soon with the advent of workspaces. More info here https://github.com/dagger/dagger/pull/11812

[normal oracle]

Are there any plans to support using a codex subscription for an llm provider? I'm building a local bug research pipeline and I had to make a codex function to run codex cli in the container instead of using the builtin llm type.

[elfin frigate]

I have this implemented on a now-pretty-stale branch, I'll get back to it when I have a chance 🙏 - it's intertwined with a bunch of other improvements to LLM, which then led to a major cross-cutting core API change, which [...]

[normal oracle]

I'm a bit confused about the split between dagger and dagger cloud. Dagger is definitely a useful tool, but it seems like it's tightly tied to the closed dagger cloud for usability. While I can run dagger locally there's little insight into the runs unless I upload everything I'm doing to a proprietary cloud. That seems to defeat the purpose of running locally. Will the dagger cloud trace/logs frontend viewer ever be open and able to run locally, or will the model always be "if you want visualization/ui build it yourself with otel or buy and upload your stuff to dagger cloud"?

[sharp marsh]

@normal oracle dagger produces OTel traces you can export to your preferred OTel platform of choice. Having said that, as you probably know, Dagger is also a VC backed company which is looking for different ways to generate revenue to keep the project going along with its open source community. As other VC backed OSS companies, there's always going to be this tension about things that will live in the OSS project, and some others that will be proprietary to the Dagger platform.

[normal oracle]

Yes, that makes sense and I understand. It seems with most products in a similar space the model is to offer "enterprise" features on top of a full featured open core. Big/enterprise customers pay and get the features/support they need while small/independent people can use and contribute back to the project, maybe with convenience cloud hosting paid options for smaller players. Dagger seems a bit different in coupling what feel like core features (at least from my initial exploration) to a closed cloud service. I'm not trying to criticize -- it just strikes me as a bit of an odd model. As a member of a tiny team I guess I'm used to it being either "this is a cloud service, we're paying for it" or "this is OSS, we're on our own unless we want to pay for support", but dagger feels a bit stuck in the middle of those two. IDK if that makes sense?

[cursive bridge]

This isn't an uncommon model. There are several other OSS projects backed by private funding/startups that follow a similar model. OSS gives the basic functionality, enterprise adds features/convienence on top of it. You COULD reproduce the enterprise functionality your self if you build on the OSS, but there's obviously a cost/benefit trade off to consider. IMO, dagger's benefits w/o the cloud offering are numerous. The enterprise features tend to favor enterprise/large scale use cases.

[sharp marsh]

Dagger seems a bit different in coupling what feel like core features (at least from my initial exploration) to a closed cloud service

Happy to discuss which of these features you think might be overly coupled to our Dagger Cloud service and look for ways to disentangle them if that makes sense. Restating my comment above in regards to telemetry, it's not coupled to Dagger Cloud if that's your primary concern. You can set standard OTEL_* env variables while calling dagger and that will let you export all the telemetry Dagger produces to your OTel collector of choice.

It's a constant struggle between DX, constant improvements and company priorities. In any case, our purpose is to make CI and E2E testing for humans and agents 10x better than any other tool out there. If a specific feature ends up being locked-in to some proprietary service we own which is not the case today, the community will definitely be vocal about that. But in the meantime, some small level coupling is needed so we can accomplish our goal. For example, the fact that you can do dagger login and immediately send telemetry for free to our cloud service for troubleshooting purposes is one of those decisions.

[normal oracle]

Yeah, I definitely could be wrong, I've only spent a day exploring building with dagger so far -- again, I'm not intending to criticize. I think dagger is a super interesting project and if I end up getting to use it more I'd love to contribute to it. I suppose in my initial exploration it feels like there's DX friction that's pushing for use of dagger cloud right out of the gate. Maybe I'm jaded by all of the data harvesting happening constantly but my initial reaction, fair or not, to "the UI is closed cloud only, you can upload your data and use it for free" was "oh, they probably want to harvest and train AI on my ci workflows".

[cursive bridge]

I feel you re: the data harvesting everywhere! ;D

Not necessary related to dagger imo, but just about everything else.

[sharp marsh]

np, no offense taken. Thx for expressing your thoughts and please keep sharing or asking whatever questions you may have. Hopefully you'll find the project useful to add value in your teams and if there's anything you're missing, happy to chat about how to make it happen 🙌

[cunning jolt]

it feels like there's DX friction that's pushing for use of dagger cloud right out of the gate
The cloud piece is entirely optional and a nice-to-have. When you start working more with dagger, you'll realize the other benefits dagger provides (like DX,UX) can take you a long way.

[sharp marsh]

TBH we mainly built it as a way to help users troubleshoot their issues and fix bugs in Dagger. It of course obviously ended up being offered as a paid tier for larger teams to collaborate

[normal oracle]

Maybe it's TUI papercuts in the initial experience? If I just run "dagger" I'm dropped into the dagger shell with afaict no help / docs beyond autocomplete and the first shortcut on the tui line is "w web" which immediately asks me to sign up for dagger cloud.

[cunning jolt]

I've had users go through this same experience. I agree that it's a bit strange that dagger drops you in the shell. I feel like the shell is an advanced use case once you get used to using dagger with dagger call

[sharp marsh]

ok, there's a bug there then 😅 . w web shouldn't appear if you're not logged in to dagger cloud via dagger login 😛

[normal oracle]

haha ok, it definitely shows up after a fresh install -- I installed dagger, ran "dagger" expecting to get standard usage text printed, and got dropped into dagger shell instead where the only thing that looked sensible for "what to do next" was the "w web" hint which immediately asked me to sign up.

I think maybe that experience more than anything made me think (again, maybe unfairly) that dagger was immediately pushing for me to send data to the closed cloud service in order to use it

[sharp marsh]

that's definitely not the intention and you feedback makes sense. I'm creating an issue so we can improve this

[normal oracle]

Also, I know from experience, users do the most unexpected (or seemingly dumb) things on first contact with your product that are super hard to see when you're working on it all the time 😆

[sharp marsh]

🙌 . My suggestion would be to start from our docs getting started page https://docs.dagger.io/getting-started (also needs improvement) which takes you through a guided path towards understanding the value of the tool.

@normal oracle I'm trying to repro your w web issue unsuccessfully.

I don't see the w web option when I'm logged-out 🤔

[normal oracle]

Oh I totally understand the value of the tool and have been through the docs, sorry if I didn't make that clear. I was just sharing my experience with the "install it and poke around" part I usually do with new tools. That's why I'm here giving feedback, because after researching tools to tame the mess of random scripts and get away from github actions dagger clearly stood out to me as the best tool/model. I'm excited by the possibilities!

[normal oracle]

Hmm, strange. IIRC after I installed dagger I created a module and set about scaffolding some functions. Some time after that is when I actually just ran "dagger" and landed in the shell. If it helps I'm on macos and installed via homebrew.

[loud briar]

For what it's worth @normal oracle we're about 12 months into Dagger, using it for image building and service deployment, and we don't use the dagger shell at all. I think it was mostly to try and make it easier to run Dagger when long commands with chains of inputs were the norm, but that's gotten easier over time and is about to get considerably easier with workspace configuration coming up

[normal oracle]

Seems like it might make sense to remove it as the default when invoking dagger without any args -- IMO just printing help text with no args is the most expected/useful behavior

[sharp marsh]

curious how to repro. If you have a sec mind running this to see if you see the w web option?

dagger logout && cd $(mktemp -d) && dagger init --sdk go && dagger

^ that should leave you in a shell session

[normal oracle]

yep, the "w web" option is present

[sharp marsh]

k.. not present in my case. 🤔 . Can you confirm you're in v0.20.8 by running dagger version please?

[sharp marsh]

k, I was able to repro now. I just needed to press esc 😛

[winter linden]

https://tenor.com/view/list-amazinggrace-court-watching-scroll-gif-13488308

catching up

[winter linden]

yeah agreed... I got over excited about the shell as single entrypoint and took it foo far (in my defense we were desperate to make Dagger more immediately useful without writing code. I really thought this would lower the bar. But there were more structural root causes)

[sharp marsh]

@neat spindle open an issue in dagger/dagger about this response/reply interaction

[cerulean token]

👋 I've struggled with the need to run KinD (Kubernetes in Dagger) quite a bit in the past. The Dagger maintainers have a k3s module, but it never starts up successfully for me. I've tried using kind's underlying node image with the same result.

I recently found the kubernetes-sig kwok: https://kwok.sigs.k8s.io/

It avoids the nested cgroup stuff that plagues the k3s module by just..not running a Kubelet. Instead, it fakes it. This solved a good portion of my use-cases, which are largely made up of Kubernetes controllers that don't actually need underlying Pods to be functional to properly test.

I have a module for it here that makes it more "Dagger-native" by using Dagger to download the Kubernetes component binaries so that they are cached. I hope that it is of use to anyone who faces the same difficulties that I have: https://github.com/frantjc/daggerverse/tree/main/kwok

EDIT: Added a rudimentary README

[sharp marsh]

awesome thx. Perfect timing as I'm currently working on fixing the k3s module as it doesn't work with kernerles without iptables support.

Regarding the kwok project, it looks interesting but I'm not sure if it'll cover our use-case given that we do atually need pods to be scheduled in our case given that we run an e2e testing of our helm chart which involves starting the engine container in a k3s cluster and validating that a client can connect to it

[cerulean token]

Yeah, I still have plenty of use-cases for running fully functioning Kubernetes cluster with an actual container runtime in various e2e tests that kwok doesn't cover. Excited to see if the k3s module changes work for me

[sharp marsh]

@cerulean token happy to help you getting it working in your case given that I'm getting muddy with it anyways. Any chance you could share a dagger cloud trace privately with an error?

[cerulean token]

As is tradition, I can't repro it now. Here's what it looked like last time I ran into the issue: #general message

Unfortunately, I did not have Dagger cloud set up at that time, so no trace. I'll circle back if I run into it again. I've got Dagger cloud set up now, so at least I'll have a trace for certain if there is a next time. Thanks!

[quiet mantle]

how are you exposing the docker socket? joined just to see if others were doing nix-built OCI image crimes with dagger 😂

very happy to see others are doing this and i don’t have to reinvent a wheel

ah lame it’s not in https://github.com/sagikazarmark/daggerverse/blob/main/nix/dagger.dang

ah, are you just mounting the local docker socket with Socket?

[quiet mantle]

do Containers, by default, have access to the entire context directory, as well as exec things with it as it’s working directory? if so, how would one disable this behavior, making a Container that does not have the (entire) context directory available?

i get the idea that this is the case from this example not mounting in anything, though please correct me if i’m wrong and this is like. a checks-specific thing or something silly like that, i dug and couldn’t find anything immediately obvious here

[winter linden]

by default every container is sandboxed with zero context access. You have to explicitly query your Workspace and inject it into the container for any contextual files to be included

[quiet mantle]

how do checks get away with not doing this, then? are they special?

(in the sense that checks take zero arguments &c)

    @function
    @check
    async def security_scan(self, severity: str = "HIGH,CRITICAL") -> None:
        """Runs security scan with optional severity filter"""
        await dag.container().from_("aquasec/trivy:latest").with_exec(["trivy", "fs", "--severity", severity, "."]).sync()

^ for example doesn’t actually map anything in

[winter linden]

no, checks are just a thin decorator on a function, registering as a hook for dagger check. You can get all the same plumbing with a regular function call dagger call

[winter linden]

this is indeed fully sandboxed

[quiet mantle]

sure, i’m not saying it isn’t

[winter linden]

ah!

[quiet mantle]

but where is the workspace being mapped in?

it’s not explicit

is dagger check doing it?

[quiet mantle]

e.g. ^

[winter linden]

you add an argument of type Workspace and the engine will inject it contextually

the type is the special case

[quiet mantle]

sure! so is this example just wrong then?

it has access to nothing from the project’s source!

[winter linden]

right, that's where the Workspace argument comes in.

[winter linden]

<@&1506565370385793125> can you modify this snippet to receive a contextual workspace and mount files from it into the trivvy container? search for the Workspace type in dagger/dagger docs and issues ("workspace api")

[quiet mantle]

sorry, i’m confused… does this just mean that the example (and the one above it, for golangci-lint) is wrong for what it’s trying to show? (linting your project via checks)

[winter linden]

unless I'm missing something, yes

[quiet mantle]

did codex dream up that default_workspace api? lol.

was gonna say… i don’t think that’s how it works

(or i would be surprised to learn so)

[winter linden]

no it actually exists but we hide it from modules for caching reasons

It's used by external clients who don't have arguments to inject into (they call dagger and not the other way around)

[quiet mantle]

isn’t it also a capability boundary? in the sense that you should never be able to get a Workspace, if you’re not a root function, and you weren’t given one

[winter linden]

yes. at the moment a root module's dependency does inherit the caller's workspace by default because it's the most convenient. But if you import an untrusted module then you might need a way to disable that inheritance. Plumbing is already there.

if you execute a container and it opens a nested client - that will NOT inherit the workspace

[winter linden]

eeh actually I guess we should disable inheritance now.

[quiet mantle]

hm, what do you mean if a container opens a nested client?

[winter linden]

dagger supports native dagger-in-dagger. so any container can (if allowed) open client connections to the engine running it

[quiet mantle]

what is the significance of the “root” in “root module’s dependency,” do modules deeper in the dependency tree behave differently?

[quiet mantle]

ah, neat. tangent: is this the only real way to do things like a function creating a tree of other functions to run dynamically?

…now that i think about it, the docs don’t mention how things like Container.exec are cached, if at all, if network access is enabled… that could lead to impurities, hm.

[winter linden]

no it's all the same, functions calling functions all the way down. The only difference is that if a module is loaded directly by a client (like your dagger CLI) it gets access to the client's workspace

[quiet mantle]

(sorry, i’m now on a second tangent.)

[quiet mantle]

ah, so everything is the same for all modules until inheritance is disabled, in which case you’ll have to explicitly pass Workspaces to everything below root?

[winter linden]

yes which is probably what you want. it's actually quite natural to pass workspace arguments, they're exactly like any other argument, so it fits in your mental model as you design your pipelines

[quiet mantle]

yep, that’s why i’m surprised that this behavior exists ;)

workspace inheritance being disabled seems like the natural choice

[winter linden]

yeah it is - just an oversight. the API is still experimental and docs not yet merged. We've been dogfooding it a lot though, and I love it 🙂

[quiet mantle]

(also, hope you don’t mind me asking all these questions… dagger seems super interesting, and very much aligns with my mental model of how CI should work as a NixOS developer 😂)

[winter linden]

not at all! I appreciate the thoughtful questions.

[quiet mantle]

(happy to move those two tangents into individual #1030538312508776540 threads if you’d prefer less clutter)

[quiet mantle]

ah, i see, https://docs.dagger.io/extending/function-caching

…though, i guess what im asking is about layer caching

https://github.com/dagger/dagger/discussions/13164#discussioncomment-16942885 implies that some steps can be nondeterministic, but i guess those steps wouldn’t be cached, and then the cached steps not getting hits, is what the issue is talking about

[winter linden]

there's also a blog post about that feature: https://dagger.io/blog/cache-control-for-modules/

[fossil pine]

It's not a magic really:

dockerSocket: Socket!,

And then in .env:

dockerSocket=/var/run/docker.sock

Finally, I use it in the code:

  pub container: Container! {
    let container = Dagger.container.from(address: "nixos/nix:" + self.nixVersion)
    let seed = container.directory("/nix")

    container = container
      .withEnvVariable(
        name: "NIX_CONFIG",
        value: "experimental-features = nix-command flakes\nfilter-syscalls = false\nsandbox = false",
      )
      .withMountedCache(
        "/nix",
        Dagger.cacheVolume("nix-"+self.nixVersion),
        source: seed,
        sharing: CacheSharingMode.LOCKED,
      )
      .withExec(["nix", "flake", "prefetch", "nixpkgs"])
      .withExec([
        "nix", "profile", "add",
        "nixpkgs#docker-client",
        "nixpkgs#docker-buildx",
        "nixpkgs#jq",
        "nixpkgs#yq-go",
        "nixpkgs#busybox",
      ])
  }

  let buildContainer: Container! {
    self
      .container
      .withUnixSocket(path: "/var/run/docker.sock", source: self.dockerSocket)
      .withEnvVariable(name: "DOCKER_HOST", value: "unix:///var/run/docker.sock")
      .withWorkdir("/work")
      .withMountedDirectory(
        path: ".",
        source: self.source,
      )
  }

I might publish it eventually, but it's still WIP.

My latest dark magic module (in progress): Rust PHP extension cross compilation 😄

[unborn pendant]

What would be an equivalent @function in python of this dagger shell command?
host | container-image "local:latest"

Because this works with images in the host docker, but dag.container().from_() doesn't.

Looks like I could somehow: https://github.com/dagger/dagger/issues/10737

[unborn pendant]

Probably I'm mixing the SDKs again... It's confusing since I think they have the same name? I'm not sure anymore. 🙂

[cerulean token]

as far as I understand, functions in modules don't have access to the host by design--you have to input everything. to that end, you could have a function that accepts a container as an input, then pass $(host | container-image local) as that input

[plain barn]

oh this is super cool, please publish it once ready!

[normal oracle]

Question about cli and args for functions -- how can I setup defaults pulled from env so I can call functions without writing an excessively long args list or adding a wrapper cli to do it for me? For example, the debug investigation pipeline I'm developing ends up taking args like this

dagger call \
  --linear-api-token=env://LINEAR_API_TOKEN \
  --github-token=env://GITHUB_TOKEN \
  --sentry-auth-token=env://SENTRY_AUTH_TOKEN \
  --codex-auth-json=file://$HOME/.codex/auth.json \
  --sentry-linear-app-installation-uuid="$SENTRY_LINEAR_APP_INSTALLATION_UUID" \
  bug-research-refine \
  --issue=ABC-123 \
  --issue-url=https://myorg.sentry.io/issues/1234/ \
  --pr-url=https://github.com/my-repo/pull/123 \
 
  --run-id=abc-123-refinement-fix-stuff \
  --codex-config=/path/to/.codex/config.toml \
  --output=.

[cunning jolt]

You can use a .env https://docs.dagger.io/features/local-defaults/. However, with workspacesV2 coming up, you will be using a config.toml (will be documented soon).

[normal oracle]

Hmm IIRC I tried that and it didn't work for secrets, i'll have to check again

[cunning jolt]

I use it for secrets. You can't put the literal secret in the .env. Anything you can specify on the CLI can go in the .env

[normal oracle]

Since i'm just getting started with Dagger should I run from source and use the new workspaces or is it not stable enough? Are there any docs yet? Will I end up refactoring a lot whenever it lands?

[cunning jolt]

I'll let the Dagger team comment on the readiness. I have plugged in some parts of the Workspaces API (which is released) into my modules but haven't fully committed yet. I would personally wait till Dagger does an official release and announcement with docs. Especially as you are new to Dagger.

I don't anticipate a ton of changes for modules I have though. The core logic of your dagger modules won't change much IMO.

[loud briar]

*dagger.Workspace args work now. Workspace config files aren't released yet

[normal oracle]

Small issue with the TUI -- first/last keys are "home" and "end". They don't exist on a macbook keyboard.

[normal oracle]

I accidentally jumped to top by pressing left arrow and now can't get back to the running part at the bottom because I'd have to hold down j or down arrow to scroll forever

[tired moth]

what is dagger-codex bot and what is it doing? how its helping dagger to solve many things and heavy lifting? is it build on top of codex sdk?

[winter linden]

we slapped codex inside a discord bot to bring more of our engineering work directly in the context of a discord thread

[viral garden]

Hey all, running into an interesting problem with generated modules clients and an external dagger host. Wondering if anyone has encountered this?

I have my dagger config with one local dependency module, testModule. I’ve also got a go generator defined.

{
  "name": "pipeline",
  "engineVersion": "v0.20.8",
  "dependencies": [
    {
      "name": "testModule",
      "source": "dagger/modules/testModule"
    }
  ],
  "clients": [
    {
      "generator": "go",
      "directory": "dagger/client"
    }
  ]
}

When I configure an External Dagger Runner host, I get the following error when trying to run the function from a go app,

returned error 422: {"data":null,"errors":[{"message":"Cannot query field \"testModule\" on type \"Query\". Did you mean \"module\" or \"currentModule\"?","locations":[{"line":1,"column":7}],"extensions":{"code":"GRAPHQL_VALIDATION_FAILED"}}]}

When I run without the external runner host, it works as expected. I’ve pinned the version v0.20.8 for both the go package and external runner image.

Anyone know if I’m missing anything? Thanks!

[normal oracle]

How can I make the TUI / traces less verbose? It seems like dagger is spitting out spans for every single file operation so the actual work my functions are doing is drowned out by a sea of withFile, withDirectory, Directory.stat and so on.

[elfin frigate]

try out --progress=report / logs / dots

we'll be making the default for non-interactive sessions less spammy soon - it's currently plain which is extremely spammy

[normal oracle]

i'm running this interactively, ideally it would be nice to just hide the low relevance common operations like file/directory stuff, there's hundreds/thousands of them in between the actually interesting withExec operations

[elfin frigate]

you can pass -q or modify the verbosity with -/+ to have it show less/more, going down a tick will likely help since it'll only show what's running

[normal oracle]

Is there a way to prevent repetitive ignore: [....] for function arguments (in typescript)? Since they have to be literals I find I'm repeating an annoyingly long list of ignores for many functions.

[winter linden]

yes there is a new API for dynamically accessing your workspace and passing ignores as code. Still experimental but about to go stable (just waiting on docs)

just receive an arg of type Workspace and the engine will inject the contextual workspace by default.

[normal oracle]

Is this part of the latest release? So far in setting up some functions I've ended up with a 2300 loc module index.ts primarily because the introspection won't allow me to refactor things to be more modular.

[winter linden]

it's been a few releases actually. we've been stress testing it on our own CI

[normal oracle]

awesome, I'll have a look and figure it out

[winter linden]

here's an example that scans the whole workspace for go packages/modules. this module takes it pretty far and even parses //go:embed directives inside the go files to add embedded files to the include filters dynamically

https://github.com/dagger/go/blob/main/go.dang#L40

[torn wren]

I want to use an AI agent to generate MR reviews in our CI pipeline. Since we are already using Dagger (dagger check), does it make sense to write a Dagger function to call the AI agent to do said review? I don't want the agent making any changes just generate a report and use that report to post an MR comment in Gitlab. If anyone has seen a blog post, guide, or repo doing something similar, please drop a link! Or if you have suggestions of how to get started I am all ears.

[torn wren]

I found this blog (https://dagger.io/blog/llm/) that describes this workflow. Am I on the right track?

// Example: Basic chaining with LLM object in Go
// Start with the LLM primitive
analysisResult, err := dag.LLM().

    // Provide its controlled environment & tools               
    WithEnv(agentEnv).     
                      
    // Give it instructions
    WithPrompt("Analyze the code...").         

    // Kick off the agent loop
    Loop().

    // Get the resulting environment    
    Env().                        

    // Extract the output
    Output("analysis_result").AsString(ctx)

[royal drift]

Hi guys, I'm looking for ways to continue learning Dagger. I'd like to know if anyone has an opportunity to work on a real project and gain experience.Thanks

[warm temple]

Hey yeah you're on the right track! If you want a working example, I have one here: https://github.com/kpenfound/greetings-api/blob/main/.dagger/review.go
I haven't run it in a bit but it should still be a good example. It's triggered by typing /review on a PR https://github.com/kpenfound/greetings-api/blob/main/.github/workflows/slash-commands.yml#L24-L30

[torn wren]

Thanks I will check it out!

[fossil pine]

I'm trying to figure out a best practice for accepting sources into functions.

The problem with workspaces is I couldn't find a way to manually create a workspace which is kind required if I want to pass in sources manually to a function.

Similarly, I'm trying to figure out where to pass in workspaces.

Here is what I came up with so far:

type Rust {
  let workspace: Workspace
  let workspacePath: String!

  new(
    workspace: Workspace = null,
    """
    Path within the workspace boundary where the Rust project (or workspace) is located.
    """
    workspacePath: String! = "/",
  ) {
    self.workspace = workspace
    self.workspacePath = workspacePath
    
    self
  }

  let maybeResolveSource(workspace: Workspace = null, path: String = null): Directory {
    let workspace = workspace ?? self.workspace

    workspace.directory(path ?? self.workspacePath)
  }

  let resolveSource(workspace: Workspace = null, path: String = null): Directory! {
    let source = self.maybeResolveSource(workspace, path)

    # TODO: switch to early return once it lands in Dagger
    if (source != null) {
      source
    } else {
      raise "workspace is required"
    }
  }

  pub foo(
    workspace: Workspace = null,
    source: Directory! = self.resolveSource(workspace),
  ): Void! @check {
    null
  }
}

• workspace is optional at the constructor level
• workspace is optional at the function level
• by default, workspace should be provided at either level, if a source is necessary
• but if a source is manually provided, neither workspace parameter is required

This allows using a module both as a toolchain AND as an individual module wrapped in other modules.

What do you think?

[loud briar]

I've done this in what I think is a pretty simple way:

(
// Workspace is automatically injected, you do not need to pass anything here
workspace *dagger.Workspace
...
// Right at the bottom of the arg list
...
// Source is for providing a remote source instead of the current workspace, e.g. from gitlab for tests
// +optional
source *dagger.Directory
) {
resolvedDir := Cmp.Or(workspace.Directory(...), source)
...
}

It's simple enough that I can live with this for now. The bit I do want is docs on pre-filtering for workspaces, which I use heavily with directories

[winter linden]

@loud briar @fossil pine the intent is that you never need a separate source directory like that. Do you need to pass an explicit workspace from the CLI, are from code?

[winter linden]

Also, it's better to avoid carrying the workspace any longer than you need it - it will "taint" every function it touches and invalidate its cache.

[loud briar]

Yeah for testing. Real use: module is called by service repository to deploy that service. Testing: module is called inside module repository pipeline and passes in other gitlab repositories which contains test services

[loud briar]

Expand on this?

What is best practice on using the workspace arg?

[winter linden]

Yeah I've run into that problem too... At the moment I run nested containers with their own dagger client.. (they get their own workspace)

but for testing I agree it would be nice to have a way to create a synthetic Workspace

[winter linden]

You can receive a workspace anywhere, but you should only use it long enough to extract immutable snapshots - then store those in fields because they have better caching.

If a function has a workspace as input, its cache will be invalidated more often than it actually needs (by design, the engine doesn't know in advance which files the function will query)

[fossil pine]

in my case, from code. the problem is if I just accept a workspace, I can't pass any "fabricated" sources to the module.

Example: I generated a rust crate from within Dagger and I'd like to build it. It's not in any workspace. How do I do that?

The other problem I see is you essentially need to pass a path to the right sources along with the workspace. Example: I want to call a module from code. I pass in a workspace, but the module also needs to know where to look for the sources it needs. it's not necessarily always the "root"

[winter linden]

in 1.0.0-beta.1 there is a dagger -W flag where you select the workspace, remote or local

[winter linden]

yes we need support for synthetic workspace, eg. dag.Workspace(myDir)

[fossil pine]

So the "best" practice is to accept workspace in a constructor?

[winter linden]

not necessarily, you can receive it directly from a leaf function or receive it from constructor. Just don't keep it in a field if you can keep materialized snapshots instead

[loud briar]

So storing workspace.Directory is likely to be better than passing around the workspace, or having multiple functions access the workspace

[cerulean token]

I am curious if anyone has found a good pattern for using Dagger modules in the "Testcontainers" pattern.

I want my tests to create a Kwok cluster for themselves from within the tests, which I already have a module for. Should the Kwok module create a client for my tests to import? Or should my dev module "own" the client?

[winter linden]

we've been adopting the testcontainers pattern from our own tests: test code orchestrate its own dependencies, using dagger. Works great. We use it in e2e and have started expanding it to core/integration (which is a big lift because of all the dagger-in-dagger...)

@neat spindle can you illustrate with links please

[tired moth]

How we can test and start adopting dagger 1.0.0-beta.1?

[winter linden]

We will cut a release and share instructions after the weekend, probably tuesday since Monday is a holiday in the US

[tired moth]

I know this question is repeating but I still want to know, it is going to support live local development and watch mode?

[plain barn]

What is the best pattern for making --fix alternatives to checks? I know normal functions cannot modify host files without an explicit export, however I am unsure if perhaps Dagger apps via dagger run could do it?

For ex I'm adding a Dagger check for https://github.com/suzuki-shunsuke/pinact
It would be nice if I could somehow also ship a Dagger app that would run the same check in --fix mode.

---

Just found out about Changeset

[cerulean token]

yeah, functions that return Changesets invoked by "dagger call" are what you're looking for. (Probably annotated with "+generate" or similar for integration with "dagger generate"--depends on the SDK)

[fossil pine]

Any chance this can make it into 0.21.0?

https://github.com/dagger/dagger/pull/13217

I think it's a serious gap for composing modules

[winter linden]

it'll have to land in 1.0.0-beta.1 because otherwise it conflicts too much with the workspace branch. We want to cut 1.0.0-beta.1 tomorrow

[fossil pine]

I think it should accept a cwd argument as well: dag.Workspace(dir, cwd: "crates/my-crate")

[winter linden]

yes agreed

[winter linden]

@fossil pine for manual workspace selection I think dagger -W is what you want. You can combine it with -m eg. dagger check -W github.com/sagikazarmark/myapp -m github.com/dagger/go -> check this workspace with that module.

[fossil pine]

My use case is still about composing modules (ie. calling one module from another).

That, combined with potentially using non-local sources (eg. Git repos).

[loud briar]

👆 Same here

I copied his test submodule repo structure 😂

[cold river]

Hello !

[winter linden]

Yo 🙂

[wraith niche]

Hey Adrien 🙂

[winter linden]

We are still testing so things will probably change before we find the right system!

[blazing sluice]

Hello ! 👋 😄

[cold river]

Alright! Glad to see you guys on Discord, you’re officially cool kids now!

[blazing sluice]

Nice Discord server icon 😏

[cold river]

Rebooting dot cloud ??

[winter linden]

lol, I wanted to put something that looked more serious

@cold river @blazing sluice do you see a "Staff" category?

[cold river]

No!

[winter linden]

Ok good 🙂

Next step: the private support channel

[cold river]

How did you do to hide it?

[winter linden]

There's a permission for it

Changed for everyone

[cold river]

Hmm... I checked everyone’s permissions on our staff channel, but can’t see that one... 🤔

[winter linden]

Those are the permissions on the Staff category

[cold river]

Oh ok! That’s global permissions for everyone, not per channel, I understand! 🙂

In our case, we do want people to see us on the voice channel while they can’t join.
They can see we do work on something... 😄

[winter linden]

ha ha 🙂

Oh, can people see the list of participants in your staff voice chan?

I see your chan, but I don't see the participant list

maybe because nobody is working right now? 🙂

[cold river]

At least nobody it talking... 😬
But yes, you should be able to see our avatars when we’re in there

We should call it “war room”

Do you see me now ?

[winter linden]

Yes!

[winter linden]

@spice valve it looks like there's no threading at all..

but @cold river is the expert 🙂

[spice valve]

That’s a pity.

[ivory nova]

Yo!

[winter linden]

Maybe it helps clarify when is a good time to move to Github issues, or some other threaded discussion tool?

In a way, Slack threading captures more information which, perhaps, should move to the GH issues sooner?

[cold river]

Yes, threading really is the missing feature in my opinion...

[spice valve]

We’re trying to get to GH discussions but so far no dice. Probably good to evaluate that first.

[winter linden]

What is GH discussions exactly? A slack competitor?

more like a Discourse competitor I guess

[spice valve]

More like a stack overflow competitor. But we don’t have it yet so nit familiar with the details. Daniel and Paul know more.

[winter linden]

I see

I personally dislike this trend of Github/Gitlab trying to do everything for everyone

But yes, that seems like a logical candidate for threaded conversation

It might help keep issues cleaner (right now issues are also a de facto user feedback forum)

[meager pier]

I like the icon for this server 😉

[patent fox]

@spice valve I'm chasing up on my GitHub contacts re discussions....

[winter linden]

@spice valve ok if I ask you a few questions here to try out the Discord experience?

You mentioned a few limitations of the new #def syntax. One I noticed is that, since "#def" forces the creation of a regular field rather than a definition, it means we can't use quotes to avoid shadowing / conflicts between definition names.

For example:

#ID: string

foo: {
   // This creates a field, not a def?
  "#ID": #ID + "-foo"
}

[winter linden]

Welcome everyone 🙂

As a reminder this is not (yet?) the official chatroom, we're experimenting with Discord as a possible replacement to Telegram

Feel free to use this room as much or as little as you want

[spice valve]

You mentioned a few limitations of the new #def syntax. One I noticed is that, since "#def" forces the creation of a regular field rather than a definition, it means we can't use quotes to avoid shadowing / conflicts between definition names.
@winter linden it would still be possible to use field aliases, though:

X=#ID: bar 
A: {
   #ID: X + “-foo”
}

To facilitate (cue) code generation, one can create an AST with preresolved identifiers and use astutil.Sanatize to automatically unshadow. (CL pending )

This can also auto-insert imports with disambiguations.

[winter linden]

Ah so one can reply and quote in Discord!

That’s actually better than slack threading I think!

[spice valve]

The problem with not having slack threading is that active discussions quickly crowd out others.

Requires some discipline from users to use it though.

[winter linden]

Yes that's true

This week we are working on our open-source strategy. What to open-source, what to keep closed, and how to make sure the open and closed parts contribute to each other's success.

[winter linden]

@cold river do you use Discord screen share to livestream code?

We're finding that text rendering is much weaker than on Zoom

But maybe we're doing it wrong?

@here we are making this Discord server the official chatroom for Blocklayer alpha. We're going to sunset the Telegram chatroom once enough people have moved over.

[winter linden]

@coarse hatch welcome 🙂

[coarse hatch]

👋

[cold river]

@winter linden no, we don’t livestream code...

[radiant crane]

hi all! a noob at discord so any tips welcome.

[gentle basalt]

yo! me too 🙂

looks pretty similar to slack!

[radiant crane]

set up a profile pic? :)

[gentle basalt]

hmm, i should really have a profile pic image lying around somewhere!

done

hmm, so no threading then?

[radiant crane]

hmm, so no threading then?
@gentle basalt I think it only has "quote-replies", like telegram

honestly no threading seems like a plus to me. it's so confusing and easy to miss.

[gentle basalt]

and a quote doesn't link back to the original message that you quoted, hmm.

[radiant crane]

I just clicked on "quote", and yeah, it only seemed to do the plaintext quoting

so telegram's quote-reply seems to be a bit better

[gentle basalt]

when there's lots of chat, things can get pretty confusing

[radiant crane]

separate channels? ;)

[gentle basalt]

yeah, just start a new channel if things are getting busy, i guess

[radiant crane]

when there's lots of chat, things will be confusing anyway... threads just hide that complexity instead of dealing with it, IMO

[patent fox]

Can you take this conversation into a separate channel, please? 😉

#roganddan

Just kidding

Otherwise I'll have to join that channel too

FOMO

That's the downside of lots of channels...

[winter linden]

welcome guys 🙂

[wraith niche]

Hello all 🙂

[spice valve]

set up a profile pic? :)
@radiant crane I think the logo is designed to encourage that. The discord logo is clearly meant to be a games controller, but as a profile pic, it looks more like a pair of boxers with buttons. Will change mine.

@radiant crane my apologies if you now cannot unsee that.

So far it seems that the Markdown handling in the desktop app is superior to that of Slack and Telegram.

[radiant crane]

Yeah, they also allow you to disable the"smart" markdown editor and just edit the plaintext.

[wraith niche]

Heads up for alpha users, we're pushing a breaking change later today. It'll require to run a bl package get -u and re-push the config (bl push <DOMAIN>).

We'll post an update here when the rollout is complete and we will ping users separately for special cases.

[spice valve]

@patent fox I actually thought you added an emoji with the discord logo.

[winter linden]

Note: the breaking change allows us to fix bugs in our Cue schema, and allow Blocklayer configurations to do more powerful things 🙂

[winter linden]

For the curious: we are tweaking the filesystem API, specifically this definition: blocklayer.dev/bl/Directory. The change is a workaround to a bug in Cue, which will be fixed soon 🙂 The bug is triggered only by Cue configurations that use a very specific pattern in a very specific way. Unfortunately our Directory met all the criteria for triggering the bug. After today's change, it will no longer meet the criteria, and therefore Blocklayer will no longer hit that bug.

The impact for you is that you can now:

• create tasks inside a for loop: (so you can dynamically create any number of tasks)
• create tasks inside a if statement (so you can run a task only if a certain condition in met in your config)

If you're using a github integration, for example to deploy something for each pull request: this will make your configuration much simpler to read and to customize.

[wraith niche]

Update: the update is live. If you have a running configuration, please run the following:

If you're using brew:

brew uninstall -f bl && brew install blocklayerhq/alpha-3/bl

Then, from the blocklayer config directory:

bl package get -u
bl push <DOMAIN>

If any issue, please ping one of us: @cloud canyon @winter linden @wraith niche

[wraith niche]

FYI, we just shipped a new feature 🚀

It's now possible to use Blocklayer locally (without touching the remote API), this is handy for a faster bl push while writing/testing a configuration from a local machine.

To use it, simply update your bl cli and run bl local up. Some instructions here: https://alpha-3.blocklayerhq.com/getting-started.html#try-out-blocklayer-locally

[radiant crane]

could you expand how trying it out locally compares to the real thing?

[winter linden]

less latency for running task. Which is very useful when you’re developing a config from scratch

[wraith niche]

and limited to your own machine capacity. On the hosted service we spawn beefy VMs (via CodeBuild), and in parallel, so it scales up with the load.

[radiant crane]

Oh, great. I was thinking more in terms of what features would be restricted to cloud, or if the local part would be open source :) I know we've touched on this before, just wondering if there are any updates given the local "engine". (Sorry for the late response, only have notifications set up on pings)

[winter linden]

Converting stackbrew to new-style definitions: https://github.com/stackbrew/stackbrew/pull/23

[winter linden]

Merging stackbrew.io into blocklayer.dev namespace to simplify developer experience: https://github.com/stackbrew/stackbrew/pull/24

[urban cliff]

👋

back at poking blocklayer today - love the "local dev" feature 😉

[wraith niche]

yeah 🙂

[urban cliff]

question: do you have a way now to publish a module? I see a bl package command but not too sure what it's for

[wraith niche]

question: do you have a way now to publish a module? I see a bl package command but not too sure what it's for
@urban cliff bl package publish is used by the stackbrew tooling for releasing packages, right now only bl package get is useful for the end user. But if it makes sense to have your package on stackbrew, I'd start with a pull-request, always eager to add more packages.

[urban cliff]

ok thanks!

[urban cliff]

pull request sent - https://github.com/stackbrew/stackbrew/pull/27

with a nice gif demo 😄

let me know if anyone is interested in this - curious - since I developed this a few months ago to manage my DNS zones

[winter linden]

@urban cliff our first external PR! Mazeltov!

[wraith niche]

pull request sent - https://github.com/stackbrew/stackbrew/pull/27
@urban cliff nice!

[late notch]

👋 is there a way to hook on a bl domain delete?
the purpose would be to fully revert what has been done on the domain. (I may have missed that part in the CLI)

[winter linden]

👋 is there a way to hook on a bl domain delete?
the purpose would be to fully revert what has been done on the domain. (I may have missed that part in the CLI)

@late notch we don't have a hook system at the moment. But we are working on resource tracking, so that every resource created by your Blocklayer config can be removed later. That could be 1) in the event of domain deletion, but also if you remove parts of your configuration, or modify the configuration in a way that requires re-creating some resources.

Another benefit of built-in resource tracking, is that it allows features like plan, where you can review the side effects of a change before making the change (similar to terraform plan for example)

[late notch]

do you have an ETA for that type of feature? (asking for a friend)

[winter linden]

No ETA yet but we have workarounds to help fill the gap. Basically annotations + custom shell scripts you can run on the side for garbage collection. We have written some of these scripts for other testers already, but they are platform-specific (one script for kubernetes, another for sql, etc).

We can share the scripts we have, or help write new scripts if needed.

[late notch]

I had some "ideas" to make a delete work with our current platform by running besides blocklayer. However if u have indeed a quick how-to I would be happy to have a look. To be "sure" to be on the good track

Was roughly thinking to list all packages used by a domain and then apply a delete (map a pkg to a delete directive) before the domain delete

As u mentioned about annotations I guess there is therefore another possible option. :)

[wraith niche]

However if u have indeed a quick how-to I would be happy to have a look
@late notch yeah we have another example with a github action that embeds the blocklayer CLI (to delete the domain), and calls a custom scripts that hardcodes several commands, such as delete a kube ns or a container image with a certain tag for instance. Resource annotations are useful to easily list resources associated for deletion.

[winter linden]

@late notch domains are meant to be very long-lived. Deleting them should be the exception and not the rule. If your workflow relies on deleting and re-creating domains often, you are likely to run into limitations.

Think of your Blocklayer as your DNS zone, or AWS account. You will probably change it often, but almost never delete it entirely.

[late notch]

This was indeed my first understanding of a domain however I have hard time to imagine the way to segment / namespace configurations related to different applications.

The original definition indeed states:

One or several inputs (optional)
A configuration (required)
One or several outputs (optional)```

However to me that transposes more to a Stack on AWS

So I was trying to find a way to have a notion of "application" (may be having several Stacks associated) so that when a command such as:

bl set -r mycompany.com env.myEnv.input.name Alice

is made it does not override the name of another "entity".

The way to do it "artificially" is by naming convention for instance stack1_env.myEnv but then no ACLs guaranty that the owner of stack2_env.myEnv will no override mine by accident by making a typo on the variable name.

I could be completely off track :/ - as you can see I try to squeeze a bit Blocklayer into an "established" paradigm and that could be in fact not how things are supposed to be managed with Blocklayer.

[winter linden]

You are not at all off track. We are still figuring out the right segmentation and you are completely right to map Blocklayer concepts to already established concepts. Our goal is for this mapping to be intuitive, even obvious. Clearly we are not there yet 🙂

Currently the closest thing to an AWS stack is what we call an environment. For example env.prod for production or env.pr for pull request review.

Note that currently environments are simply a configuration pattern. You can actually organize your configuration any way you want. The env.<envname> layout is just a recommendation. That may change in the future once we settle on the right nomenclature.

One difference with aws stacks is that Blocklayer environments can be configured and even generated dynamically by the domain configuration.

For example you can iterate over a map of pull requests and associated properties (git url, branch name etc) and generate a map of environments, one for each pr.

Or alternatively, you can have one static environment for all PR deployments, and dynamically generate that domain’s configuration from the map of PRs.

Both patterns are possible, we’re not sure which one is better yet. I am partial to the latter (static environments with dynamic contents) but it’s subjective.

I hope this helps!

[winter linden]

On your point about env ACL: you are correct and we plan to add that feature so that you can control who can access which env.

[late notch]

Understood.
I think what is tedious to accomplish at the moment is to get the idea of multiple configurations belonging to a domain. For instance if I am not mistaken bl domain history would give you the entire configuration history but it is (I think) impossible to get this big chunk of configuration detailed by application.

Using package name + file name could be a possibility. Such as my application Foo is in 2 separated repo because made of 2 services svc1 & svc2 . Those 2 services are part of Foo. Foo is one application among many within the domain bigcorp.com
svc1 has its own config as well as svc2.

Therefore that would be bl push bigcorp svc1.cue and bl push bigcorp svc2.cue assuming that the users did not copy paste the config of each other by naming their package package svc1 with no intention to override the later.

Also doing bl history would not give me an explicit view about what changed in svc1 and svc2 for the app Foo.

That is why I though using domain as namespace, because it garanties the segmentation of each application. In that particular case Foo would be a domain. The segmentation of configurations between svc1 and svc2 within Foo would be a matter of generating those configurations on behalf of the user to not have unwanted override. So a delete for instance is only referring to all the configuration of a domain. A delete of svc1 only would be still a bit hard but possible with a bit of external logic.

I apology in advance if what I just exposed is already answered above. If it is the case, I think I need to practice more Blocklayer to really have a grasp at the capabilities in term of configuration segmentation or I need a concrete example illustrating a multi applications (each one having multiple configs because of multiple layer of services - like the acme example) 🙂

[winter linden]

This is very relevant to our next iteration of the design! Maybe we could give you a preview next week and get your feedback?

[winter linden]

First PR merged! Thank you @urban cliff !

[late notch]

@winter linden with great pleasure! 🍿

[spice valve]

Please note that cue v0.2.1 is now out. This release is quite forceful in deprecating some of the old language features. For instance, :: is no longer allowed. It is still supported under the hood, so you can still run cue fix. This is mostly done to bring CUE to a clean slate before swapping out the new evaluator.

[winter linden]

Thanks @spice valve that's good to know. We still have to port stackbrew packages to the new definition syntax. We tried last week but it was more work than we expected, we had to postpone.

The longer we wait the more painful it will get... We will try again soon.

[spice valve]

If the v0.2.1 changes are too aggressive I can revert. But better not.

[winter linden]

It was nothing insurmountable. Just tedious.

[spice valve]

Did you run into cases cue fix (not cue fmt) couldn't handle? If there is a specific case it doesn't support I can possibly automate that.

[winter linden]

I don't think they're too aggressive for us, we can stick to the currently pinned version for a while

Did you run into cases cue fix (not cue fmt) couldn't handle? If there is a specific case it doesn't support I can possibly automate that.

Yes we got quite a few of the tmpXXX comments

[spice valve]

Also note that I've updated the LookupDef API to automatically insert a #. A bit nasty, but it can be done unambiguously. If there are other cases like that let me know.

[winter linden]

some false positives on the reference guesses

[spice valve]

Interesting. If you can send some the things that would be nice to have a fix for I can see if it can be done easily.

[winter linden]

I ran into one issue which may or may not be caused by cue fix, or a breaking change in shadowing behavior

that one I remain stuck on

We can share a clean repro if it helps

(or at least try)

Another issue is that cue fix skips _test.cue files

And we have a lot of those

[spice valve]

Up to you. If the effort to fix the configs is about equal, then better do that. There is no guarantee there is a fix.

Ah, the _test.go can be fixed. It also doesn't handle multiple packages in a single directory. That is a harder one to fix. We'll fix it eventually, but probably don't want to hold up the new evaluator for that.

But yes, it turns out that static resolution is incredibly hard. Another data point that the switch to using # seems to be a good one! It also simplifies the new evaluator a bit. That is why the old :: is no longer supported.

[winter linden]

We don't use multiple packages per directory, so we're all good there

Thanks for the update! I'm off to bed.

[spice valve]

Nighty night!

[spice valve]

I've submitted https://cue-review.googlesource.com/c/cue/+/6440, which now includes _test.cue and _tool.cue files for cue fix.

[winter linden]

👍 thank you!

[spice valve]

BTW, the fix functionality is now also available as an API in tools/fix.

[ruby spoke]

👋 glad to be there!

I ran into an issue pushing my first config. It seems to be after downloading blocklayer.dev/bl

info Downloading blocklayer.dev/bl
fatal expected operand, found 'for':
    ./cue.mod/pkg/blocklayer.dev/bl/bash.cue:42:21
string literal not terminated:
    ./cue.mod/pkg/blocklayer.dev/bl/bash.cue:43:38
illegal character U+005C '\':
    ./cue.mod/pkg/blocklayer.dev/bl/bash.cue:46:4
string literal not terminated:
    ./cue.mod/pkg/blocklayer.dev/bl/bash.cue:47:15
string literal not terminated:
    ./cue.mod/pkg/blocklayer.dev/bl/bash.cue:49:4
string literal not terminated:
    ./cue.mod/pkg/blocklayer.dev/bl/bash.cue:51:2

Any idea on what I'm doing the wrong way ? Oh and btw the blocklayer.dev/bl checksum is 8be9d4b7f7ae807ec4bff7502704096437101240aadf18690b2b44d2b2efa9d2

[wraith niche]

hey @ruby spoke, I think there might be an issue with your config that causes this error. Can you share your whole config ? Either here or privately.

[ruby spoke]

Nervermind, the issue did not appear on my laptop. Must be something broken on my desktop config.

[ruby spoke]

Did somebody manage to deploy a kube config on the docker-desktop cluster ? Currently I'm facing :

Unable to connect to the server: dial tcp: lookup kubernetes.docker.internal on 8.8.8.8:53: no such host

Which to be honest doesn't surprise me at all

If anybody as already found a work around it would be nice. Otherwise I'm gonna try some tricks to tweak the dns lookup

Oh and of course that's only for a local instance of bl, wouldn't make any sense otherwise

[winter linden]

It looks like we might need to tweak the network settings of the bl-buildkitd container

maybe try pointing to the IP address of the kubernetes endpoint instead of its hostname? It might be a DNS-only issue

(I’m just guessing, haven’t tried deploying to the local kub cluster)

[ruby spoke]

Yup that make sense, I’m gonna try something like that tomorrow, I keep you updated

[cunning yacht]

Hi guys, I prefer discord

[ruby spoke]

@winter linden yup it's working with the kube apiserver ip

bl set <domain> env.kubernetesDeploy.input.kubeconfig.value \"(sed -e s/kubernetes.docker.internal/(kubectl get pod kube-apiserver-docker-desktop --namespace kube-system --template={{.status.podIP}} )/g ~/.kube/config | base64 )\"

(you may want to add some $, I'm working with fish shell)

Maybe, if I'm not the only one doing some weird local stuff, we could add a localConfig package embedding the logic (like aws/googlecloud.Config)

[winter linden]

I'm sure it would be useful to others!

[ruby spoke]

Dumb questions (i'm gonna have a lot of them I think 😅 ):
1- Is there a way to customise the docker-compose used by bl local up (I know I could redeploy a stack manually and I will eventually to add a traefik w/ a local dn, but I'm a lazy guy) ? Mainly to mount volumes and stop claiming the same domain each time I restart the local stack.
2- When exploring the graphql api (http://localhost:8080/query), I get a 401. I've tried to generate an api-key and use it as a bearer token in headers without success. I'm sure I missed something but I don't know what.

3- Is there something like terraform destroy on bl ? Currently I'm just destroying the namespace manually.

[winter linden]

Hi @ruby spoke ! Thanks for the detailed questions. Replies coming up.

1- Is there a way to customise the docker-compose used by bl local up (I know I could redeploy a stack manually and I will eventually to add a traefik w/ a local dn, but I'm a lazy guy) ? Mainly to mount volumes and stop claiming the same domain each time I restart the local stack.

That probably requires us to patch the cli for you. But it shouldn’t be too hard. Will check with @wraith niche who implemented that feature.

2- When exploring the graphql api (http://localhost:8080/query), I get a 401. I've tried to generate an api-key and use it as a bearer token in headers without success. I'm sure I missed something but I don't know what.

The way you tried it is correct. You’re probably 99% there. Can you share more details on how you’re generating the API key and using it?

3- Is there something like terraform destroy on bl ? Currently I'm just destroying the namespace manually.

Not yet but we are working on it :)

We do have helper scripts to semi-automate namespace destruction, as a stopgap. We can share those to save you some time.

[ruby spoke]

Thanks for your responses !

The way you tried it is correct. You’re probably 99% there. Can you share more details on how you’re generating the API key and using it?

bl apikey create browser

Inside the Headers part of the graphql playground (localhost/8080) :

{
  "Authorization": "Bearer the_token"
}

Not yet but we are working on it :)

We do have helper scripts to semi-automate namespace destruction, as a stopgap. We can share those to save you some time.
Yup I can make use of those scripts, thank you!

[wraith niche]

1- Is there a way to customise the docker-compose used by bl local up (I know I could redeploy a stack manually and I will eventually to add a traefik w/ a local dn, but I'm a lazy guy) ? Mainly to mount volumes and stop claiming the same domain each time I restart the local stack.
@ruby spoke do you want to persist the state of the local api? Replacing the compose file is not so hard but you'll have to re-do a lot of things yourself. If it's just the API state (domains etc...), we can find a way to export / import it cleanly.

2- When exploring the graphql api (http://localhost:8080/query), I get a 401. I've tried to generate an api-key and use it as a bearer token in headers without success. I'm sure I missed something but I don't know what.
@ruby spoke the right header is X-Api-Key: <token> when you use a custom api key for the graphql API

btw, no dumb question 🙂

for the API key, you can see an example of the header by running:
bl --api-key foobar --debug domain list -> this will fail but shows the right header

the graphql api is internal though and can break. No problem to use it from scripts, just a heads up. If you need specific features from the CLI, we can prioritize some for you, just let us know.

[ruby spoke]

Yeah don't worry that was just me being curious 😅

We're actually working with graphql on our side too

[wraith niche]

no problem, if you want to chat gql someday, @cloud canyon spent quite some time spec'ing this API

[ruby spoke]

@ruby spoke do you want to persist the state of the local api? Replacing the compose file is not so hard but you'll have to re-do a lot of things yourself. If it's just the API state (domains etc...), we can find a way to export / import it cleanly.
Yeah that's mostly for the API state. I was also thinking of mapping it to my local traefik instance but that was just to have a blocklayer.host endpoint instead of mapping to 8080.

@odd wigeon is our gql wizard, I'm sure he would love that when he manage to find some time 😅

[ruby spoke]

Hello there, not sure if it's cue or bl related question. I'm trying to fill a field with the content of a file.

configs: [{
  mountPath: "/docker-entrypoint-initdb.d"
  kube: {
    metadata: name: "postgres-init-cm"
    data: "postgres-init.sh": string // Here read from a local file
  }
}]

Any idea what the best way to do that ? What I find to be the best solution right now is to get this data as an input, create a draft with the content of the file (and all similar configs) and then push this draft. I found it not perfect since this is a config map in the end. If it was a json/yaml file I would just have import it in cue. Another solution would be to write the content of the file as pure string, but that's kind of ugly IMHO. I've also tried to use cue std "tool/file" but I'm not sure if/how I can link a task output to definition field / bl input.

[winter linden]

Hi @ruby spoke, what’s the ideal lifecycle of this file? Will it be updated by the same people updating the blocklayer config? Or will it sometimes be updated separately?

If it’s the former (update together) then you can push the file alongside the config as an attachment, then access it with the blocklayer filesystem API.

If it’s the latter (update separately) you can either:
a) push the contents of the script as a string
b) push a scripts directory as a bl.Directory (if you might have multiple scripts)
c) have your blocklayer config fetch the script from its git repo

Your idea of using the standard cue tasks + tool/file makes sense, but blocklayer does not support that style of tasks. We have discussed the option, but no short-term plan to support it at the moment.

[ruby spoke]

Thanks! I didn't think of the bl filesystem API, that could be a solution as this is the local deployment. a) is what I was doing. c) is not an option for that use case since if we change it it will be locally (but that's a totally viable solution for our normal use case). I'm really curious about b), but for another use case. Will it be possible to "mount" the content of a bl.Directory as a kubernetes volume in any way ?

[winter linden]

Will it be possible to "mount" the content of a bl.Directory as a kubernetes volume in any way ?

@ruby spoke I think so - how would you do it normally? If you can do it from a regular container, normally you can do it from blocklayer.

[ruby spoke]

As a dirty hostPath 🙈

[winter linden]

well setting up the hostPath shouldn't be a problem, but your blocklayer config is sandboxed and doesn't have access to your host filesystem... So you would have to get it there somehow. For example you could ssh into the host and rsync the files?

[ruby spoke]

Well it's just for the local deployment (some dirty test files are host mounted, relics of our docker-compose way of doing). So I think in that case it should be wiser to just pass the pwd as an input and use it in the kube conf generated.

[winter linden]

Ah I see. Yes that can work too.

[ruby spoke]

2020, when it's harder to deploy locally than in the cloud 😅

[winter linden]

🙂

How will you mount that directory in non-local deployments?

[ruby spoke]

will not

That's just some testing files. Stateless ftw

[winter linden]

ah I see

For initializing the DB right?

[ruby spoke]

Oh no, this one is not a problem, it's mounted as a configMap.

But yeah you're right in a way: our db in the cloud is managed, and not in kube obviously

I'm talking about binary files on some (actually one) of our service

[winter linden]

code or data?

[ruby spoke]

But it's a flaw in the architecture, our webservice should not be stateful for random reason

Data

[winter linden]

how large is it, roughly? Less than 1GB?

[ruby spoke]

Some space packets IIRC

yup

[winter linden]

I don't know what a space packet is, but it sounds very cool 😉

[ruby spoke]

10Mb for the big one

Ahaha i don't know either tbh

[winter linden]

You know, for a 10Mb file, it might be better to just build a container with the file inside. Then you have full parity between local and cloud deployment, and that file goes through the blocklayer storage layer, so you get full history, snapshots etc

(just a thought)

[ruby spoke]

I'm gonna screen that and repost it in the thread I asked for this change

Thanks for the (unintentional) support 😂

[winter linden]

Or, if there is resistance to building the data file into the container, you could push it to a S3 bucket or equivalent, and then inject the URL to the container so that it can download it

Then, on local deployment, you can replace the S3 bucket with a local static http container

[ruby spoke]

Yup, told that too

[winter linden]

😄

[ruby spoke]

Glad we're aligned

To be totally honest the team is ok to do so, but that's not the priority right now (aka I will do the refactor at some point) and I can understand that

[winter linden]

Makes sense.

Ideally blocklayer would make it easier for you to implement the best pattern, without requiring the dev teams to do any work, or change their habits

[ruby spoke]

Yup, it already does actually

[winter linden]

! good to hear. Anything that you think would make that easier, let us know. It's a priority for us to adapt to existing workflows.

[ruby spoke]

Yup, will do. But we are also in the path to build the best workflow, so it's a both way work

[ruby spoke]

Quick question: does the bl push domain --draft d -k secret is implemented ? As for now it yelling me fatal open secret: no such file or directory and I'm not sure if I did something wrong. I read in the doc that you need to actually set it using -k text and the .value path so I assume -k secret is not implemented yet, right ?

https://alpha-3.blocklayerhq.com/domain.html#stage-domain-changes-in-a-draft

[wraith niche]

Hey @ruby spoke, it's implemented in the draft but probably a bug. Let me check.

Yeah there is an issue with the target on that command. Fix on the way!

[ruby spoke]

Dope !

[wraith niche]

@ruby spoke we just released a new CLI to address this issue. The root cause is that --target is optional with bl push and defaults to ., which does not make sense for a secret key. So we added an error check for this. The correct syntax to use a secret in a draft is bl push domain --draft d -k secret -t input.myKey (no need to pass a .value). We usually just use bl set -s domain key for simplicity.
If you upgrade your cli, you should have the fix, thanks for reporting 🙂

[ruby spoke]

Thanks! Gonna update tomorrow

[ruby spoke]

Another issue (but I have the feeling this one is on me). I'm trying to read a file using the package stackbrew.io/file :

foo: {
  f: file.Read & {
    filename: "foo.bar"
  }

  output: f.contents
}

I probably misread the doc since:

foo.f.script        | task failed: failed to resolve task inputs: bl.Directory "input[\"/src\"]" is not concrete 

should I override the source ?

source: bl.Directory & { source: "context://<???>" }

looks promising since it give me a file not found when using random path and rpc error when I using root

Guess I need to set the proper workdir context

[ruby spoke]

Another one, is there any simple way to use gcr.Credentials as an ImagePullSecrets for kubernetes pod ?

[wraith niche]

source: bl.Directory & { source: "context://<???>" }

looks promising since it give me a file not found when using random path and rpc error when I using root
@ruby spoke indeed you need to set the source. It can be either a bl.Directory that you directly upload from the CLI (bl push domain -k directory -t input.source ./my/path)

or any other bl.Directory. For instance:

repo: git.Repository & {
  url: "..."
}

source: repo.out

Another one, is there any simple way to use gcr.Credentials as an ImagePullSecrets for kubernetes pod ?
@ruby spoke gcr.Credentials will only be the auth for bl.Push / bl.Pull. But I guess you can give access to GCR from the kube cluster role so you don't have to provide creds? Maybe I'll have to look at your config for that one...

[ruby spoke]

Yep don’t worry I have a way to do it, I was just wondering

Thanks for pointing me on the right direction for the directory

I’m curious about the context// tho, what does it represents ? And the registry is a docker registry ?

[wraith niche]

it's an internal, it represents the context for buildkit (used by the bl runtime), but ideally it would be invisible from you (and likely to be in the future).

ideally you should simply manipulate bl types (like bl.Directory) and this kind of value isn't needed. But if you have limitations, let us know. It's always possible to add missing features, or new packages on stackbrew.

[ruby spoke]

Ok i get it thanks! No limitation yet, I’m just curious 😅

[winter linden]

@ruby spoke just FYI, we have simplified the Blocklayer stdlib by merging stackbrew.io and blocklayer.dev into a single namespace. The recommended import is blocklayer.dev

In practice stackbrew.io is aliased to blocklayer.dev for backwards compat. But that may change in the future.

The idea is to make stackbrew.io the bleeding edge upstream, and blocklayer.dev the stable, officially supported downstream

(right now they are the same)

[ruby spoke]

Massive sed incoming 😅

Thanks for noticing

[winter linden]

I just did the same sed on the docs

🙂

don't forget bl.sum

[ruby spoke]

Yup, if I delete and repush it that would do the trick right ?

[winter linden]

yes

(🤞 )

[ruby spoke]

@wraith niche I didn't manage to get the bl directory working. I think I'm missing something. Here is what I pushed:

package main

import (
    "blocklayer.dev/bl"
  "blocklayer.dev/file"
)

foo: {
  input: configDir: bl.Directory

  f: file.Read & {
        source: input.configDir
    filename: "foo.bar"
  }

  output: f.contents
}

and then bl push my.domain -k directory -t input.configDir ./configs/

And here is the result:

success Upload complete. Digest: sha256:e164eacc3390bc50c5a86de0dce802a8aa7a8b0ed3dfbe71fb3bf6d3f34eebee
success Pushed new change with ID 6d006ce2-e2e6-46b5-ac79-dc04628e67a5
info Waiting for the change to be scheduled...
foo.f.script.build  | running task
foo.f.script        | waiting for dependency to complete    dependency=foo.f.script.build
foo.f.script.build  | task completed    duration=200ms
foo.f.script        | running task
foo.f.script        | task failed: failed to resolve task inputs: bl.Directory "input[\"/src\"]" is not concrete    duration=0s
system              | failed to resolve task inputs: bl.Directory "input[\"/src\"]" is not concrete

My bad I'm an idiot. I forgot the root path (foo) in my target 😅. Works totally fine

[ruby spoke]

Hey @wraith niche which version fixes the bug on the secret for draft ? I'm at f36b851e and :

bl push domain --draft k -k secret -t env.foo.input.secret "bar"
fatal open bar: no such file or directory

[wraith niche]

Hi @swift barn, you have the latest version. Here is the brew formula: https://github.com/blocklayerhq/homebrew-alpha-3/blob/master/bl.rb

"bar" expects a file, useful if you want to specify the content of the file as a secret. Just remove "bar" at the end and it should prompt you on the cli for the value.

let me know if it works, the help is misleading...

[ruby spoke]

Oh thanks! I see the logic in that but didn't understood at first sight

I replaced brew for macPorts some time ago (totally personnal opinion: poor architecture & kinda toxic community) but the curl is fine for me.

[wraith niche]

I am fine with any kind of binary install 🙂

[ruby spoke]

Ok I managed to reproduce the bug with a simple configuration I'm comfortable to publish here

package main

import (
    "strings"

    "blocklayer.dev/file"
    "blocklayer.dev/bl"
)

env: config: {
    input: dir: bl.Directory

    f: file.Read & {
        source:   input.dir
        filename: "foo"
    }

    l1: [f.contents]
    l2: [ for x in l1 {x}] // Should be same as l1

    s1: strings.Join([ for x in l1 {x}], " ")
    s2: strings.Join(l2, " ")
    s3: strings.Join(l1, " ")

    task: bl.BashScript & {
        environment: FILE_CONTENT: s1 // Fails
        // environment: FILE_CONTENT: s2 // Fails
        // environment: FILE_CONTENT: s3 // Works
        code: "echo Hello $FILE_CONTENT > output.txt"
        output: "/output.txt": string
    }

    output: {
        content: task.output["/output.txt"]
    }
}

[spice valve]

Sorry, missing context. Is this a CUE bug or bl bug?

[ruby spoke]

@winter linden it indeed seems like it's related to the list comprehension

Oh yeah sorry I started the discussion in our private channel since I was not sure I could publish our config right here. Not sure if it's cue or bl related, I'm gonna try with pure CUE but I think the fill works in CUE. The issue here is bl not resolving the dependency between task and file read, when doing some wingardium leviosa with list comprehension.

[spice valve]

It could be that dependency analysis doesn't work well for list comprehensions. In CUE you can also specify dependencies explicitly. Not sure if that works here. Reverting to Solomon. 🙂

[ruby spoke]

Yup, he told me that @cloud canyon already worked on a similar issue, so it's probably related in some way

In CUE it works if I embed the list inside a task:

package tool

import (
    "strings"

  "tool/file"
  "tool/cli"
)

command: config: {

    task: f: file.Read & {
        filename: "../dir/foo"
    contents: string
    }

    task: print: cli.Print & {
        text: s1

        l1: [task.f.contents]
        l2: [ for x in l1 {x}] // Should be same as l1

        s1: strings.Join([ for x in l1 {x}], " ")
        s2: strings.Join(l2, " ")
        s3: strings.Join(l1, " ")
    }
}

I cannot do the same tricks in bl since the struct are closed

actually it could be as simple as that :

package main

import (
    "blocklayer.dev/file"
    "blocklayer.dev/bl"
)

env: config: {
    input: dir: bl.Directory

    f: file.Read & {
        source:   input.dir
        filename: "foo"
    }

    task: bl.BashScript & {
        environment: FILE_CONTENT: [ for x in [f.contents] {x}][0] // Fails
        code: "echo Hello $FILE_CONTENT > output.txt"
        output: "/output.txt": string
    }

    output: {
        content: task.output["/output.txt"]
    }
}

[ruby spoke]

And a little workaround to trick the system dependency management:

...
task: bl.BashScript & {
        environment: {
                    WINGARDIUM_LEVIOSA: f.contents
                    FILE_CONTENT: [ for x in [f.contents] {x}][0] // Works
                }
        code: "echo Hello $A > output.txt"
        output: "/output.txt": string
    }
...

[ruby spoke]

Is there a way to manually set dependencies between task ? Nothing related to the previous issue. But given task1 and task2. task1 deploy a kube secret and patch the default SA to use this secret as a image pull secret. task2 create some deployment that contains images from private registry. Currently the pod goes in ImagePullBackOff and I have to delete it in order to take the SA change into account.

[winter linden]

you just need to create a reference from any value in task2 to any value in task1. For example it could be the status field

[ruby spoke]

Ok makes sense

[winter linden]

(this also works in regular cue tasks)

@ruby spoke here's the bl.Run reference https://github.com/stackbrew/stackbrew/blob/master/pkg/bl/bl.cue#L68

it embeds the bl.Task primitive which is defined here: https://github.com/stackbrew/stackbrew/blob/master/pkg/bl/bl.cue#L38

[ruby spoke]

Yup, that's the same kind of trick I used to workaround the list comprehension issue

Works like a charm, thanks!

[winter linden]

We're finally updating to the new Cue definition syntax, introduced in Cue 0.2: https://github.com/stackbrew/stackbrew/pull/29

Once merged, this will introduce a breaking change to your configurations, but:

1. The breaking change only occurs when you update your dependencies (bl package get -u)
2. It's very easy to upgrade your configuration: just replace Foo with #Foo, where Foo is the name of the definition you use from the blocklayer standard library
3. If you updated your dependencies but don't want to deal with the fix right now, you can simply revert bl.sum to its previous contents, and push again.

We will let you know as soon as we publish the change. Probably tomorrow.

[spice valve]

Note that the new evaluator (v0.3.0), will no longer support the old-style definitions. The plans is for this to be the last large breaking change to the language. A few more that are being considered (like making float equal to numberare relatively non-consequential and mostly just a relaxation of the language. So v0.3.0 will be a big step towards a go-style backwards compatibility guarantee.

[ruby spoke]

Hi there! One of our users is experimenting an issue on his local blocklayer environment : fatal Post https://localhost:8080/query: http: server gave HTTP response to HTTPS client. I'm not able to reproduce it on my setup, and we are on the same bl version (f36b851e). Does that ring a bell to you ?

[wraith niche]

Hey @ruby spoke! I would guess the BL_API_SERVER env var is set to https insteand of http

when you run bl local up, you just have to copy/paste the export, here is an example:

[ruby spoke]

Bouahahaha I didn't even think to check that. Sorry for that question 😅

[wraith niche]

no worries, we wanted to make this seemless (without the env var) for this exact reason 🙂

[muted gale]

Hello!

I wanted to follow up on our call today about some relevant writing we discussed:

https://blog.cedriccharly.com/post/20191109-the-configuration-complexity-curse/
This is when I started writing about CUE. Since you all are already familiar the last third of the essay is where I lay out the long term vision. I think that part is relevant to Blocklayer for building out an ecosystem that is beneficial to the community while driving value back to the service.

https://blog.cedriccharly.com/post/20200426-kubernetes-the-universal-control-plane/
I lay out what I value the most out of the Kubernetes (the resource model) why I think that and control planes are powerful tools for infrastructure and service management. That is relevant to IaC because...

https://github.com/cuelang/cue/discussions/431
This describes why I want to use CUE for IaC and what that could look like in the future. See the outline in comments where I break down the individual parts of IaC tools. Blocklayer can be an advantage as part of the workflow

[winter linden]

Welcome @muted gale ! Thanks for the reading material. It's definitely very relevant to Blocklayer.

[muted gale]

Yes, let me know what you think