#maintainers

1 messages Β· Page 20 of 1

tepid nova
#

There should be a stable target version, that determines what you generate - then either a given engine can provide that view, or it can't

civic yacht
#

Yes I agree, that's what Tibor was suggesting I think, use the engine version in dagger.json

tepid nova
#

Ah yes, exactly πŸ™‚

#

So, does this mean that the current state of main was generated with a main engine, which was required for CI-on-main to pass, and now everyone developing their branch with 0.20.8 is reverting that codegen every time we run 'dagger generate'?

#

So in other words, for my PR to pass CI, I have to install a main build of dagger

#

Very sneaky way to force everyone to dogfood, I like it

charred lotus
#

sorry, in the meantime:

curl -fsSL https://dl.dagger.io/dagger/install.sh | DAGGER_COMMIT=head BIN_DIR=... sh
civic yacht
#

Hm. we appear to be getting rate limited trying to obtain the token that we use to not get rate limited by dockerhub? if I'm parsing this correctly?

failed to fetch anonymous token: unexpected status from GET request to https://auth.docker.io/token?scope=repository%3Alibrary%2Falpine%3Apull&service=registry.docker.io: 429 Too Many Requests
``` ([from here](https://dagger.cloud/dagger/checks/github.com/dagger/dagger@47ffb4fcb7a777930118b6178e06e183688d8f30?check=typescript-sdk:test-nodejs-lts&viewMode=trace#1cfebb13d540ad54:L1814))
#

Or does that mean we aren't even using our token at all?

charred lotus
#

failed to fetch anonymous token:

#

looks like token isn't set up properly ?

civic yacht
#

(according to the robot, yes that's how it works)

civic yacht
#

so yeah I guess our token is just missing somehow

civic yacht
kind hearth
#

hey; i want to contribute to Dagger, want to know if a Linux machine or VM is required for development, or macOS is sufficient for most of the work. What is the recommended local setup for contributors?

tepid nova
still garnet
#

I'm seeing this failure a lot (trace):

❯ ~/src/dagger/better-tests/bin/dagger check
βœ” load module: go 0.4s
βœ” load module: midterm 0.4s
Error: loading modules: loading module "github.com/dagger/go@main": resolving module source "github.com/dagger/go@main": failed to load git dir: zjcrh039ljmw4is4s66bm0zv7: not found

When it happens it's always triggered by usage of github.com/dagger/go - I've observed this in dagger/dagger and now in https://github.com/vito/midterm (where I just adopted it and it's working great otherwise btw - living the drop-in toolchain dream)

I'm running my own branch's CLI + dev engine above so there's a chance it's my own fault, feel free to ignore until I have a repro on main, just raising in case someone else is hitting it too

EDIT: can repro easily on main against https://github.com/vito/midterm - see #1504168990279205055 message

civic yacht
#

I taught codex how to trigger re-runs in CI (using my auth token from my homedir), let it run overnight to try to shake flakes out, created this nice looking trace summary page https://dagger.cloud/dagger/checks/github.com/dagger/dagger@2be47918021e5b05a4ca422c8c09f36761a3b0e5

I never triggered the original flake I wanted, but did hit some other interesting ones (besides uninteresting 500 errors from external network deps). Very tempting to just let this run indefinitely in a loop and hand legit looking errors off to a human and/or agent thinkies

charred lotus
still garnet
tepid nova
#

@still garnet does dang SDK give me direct access to loadFooFromID?

tepid nova
tepid nova
#

@still garnet I think I may be hitting the limits of Changeset + Workspace...

https://github.com/shykes/dagger-go-sdk

Example:

$ cd ./my/workspace

$ dagger -m github.com/shykes/dagger-go-sdk call mod --path=./path/to/a/go/module generate
 Apply changes?
┃ toolchains/engine-dev/dagger.gen.go                     +1136
┃ toolchains/engine-dev/go.mod                            +10 -10
┃ toolchains/engine-dev/go.sum                            +22 -20
┃ toolchains/engine-dev/internal/
┃ toolchains/engine-dev/internal/dagger/
┃ toolchains/engine-dev/internal/dagger/alpine.gen.go     +322
┃ toolchains/engine-dev/internal/dagger/codegen.gen.go    +152
┃ toolchains/engine-dev/internal/dagger/dagger-cli.gen.go +348
┃ toolchains/engine-dev/internal/dagger/dagger.gen.go     +16008
┃ toolchains/engine-dev/internal/dagger/go.gen.go         +768
┃ toolchains/engine-dev/internal/dagger/notify.gen.go     +225
┃ toolchains/engine-dev/internal/dagger/version.gen.go    +239
┃ toolchains/engine-dev/internal/dagger/wolfi.gen.go      +183
┃
┃ 13 files changed, +19413 -30 lines
┃
┃                       Apply     Discard

✘ applying changes 0.0s ERROR
✘ Changeset.export(path: "."): String! 0.0s ERROR
! internal error: Directory.diff received different relative paths: "/before" != "/after"
civic yacht
#

working through a bunch of fix and improvement PRs currently getting them solid, but I think this one crossed the threshold now https://github.com/dagger/dagger/pull/13123

reduces allocations a containerd boltdb writes a ton by just getting rid of unnecessary lease creation

charred lotus
civic yacht
#

agree it's needed for the release, but yes worst case scenario if we just fallback to "private == locked" for now that'll suffice

charred lotus
tepid nova
#

@still garnet in my module I execute a helper tool that prints out include filters to stdout. Completely internal, not meant for users to see. But with dagger check it is prominently shown to the user as "logs". Is there an otel attribute incantation to prevent that, without breaking other things?

#

Love a nice readable error message:

failed to resolve dep to source: failed to load sdk for dir module source: invalid SDK: "unknown" [traceparent:d905b195ff 456a765cca663a0618-8ad7cabfcbac0874]

πŸ€”

still garnet
still garnet
# tepid nova Love a nice readable error message: `failed to resolve dep to source: failed to...

if the bulk of the complaint is the [traceparent:...] part, lemme know where that's popping up and I'll strip it out if possible. The downside of propagating error origins through strings is having to play wack-a-mole in the UI, but at least it's super durable and there are finite places in the UI to clean it up. (It can still end up showing up places like test logs, but that's saved my ass sometimes when it shows up)

tepid nova
tepid nova
#

I guess I can change to writing it to a file

tepid nova
civic yacht
#

while poking around go tool trace of the engine I noticed we were blocked for like 300ms before each container invoking CNI, which wasn't expected since there's supposed to be pooled re-use of CNI netns'es. Turns out that got accidentally disabled in effect at some point due to a default hostname being set on each with exec.

Fixed that, TestModule takes 30% less time to run for me locally now: https://github.com/dagger/dagger/pull/13144

tepid nova
#

That's on every single function invocation too then?

civic yacht
tepid nova
#

Oh yes! For sure!

#

That's crazy. Awesome find

wild zephyr
#

the theseus release is becoming more and more appealing now.

tribal mica
#

Hey, I made a PR to add CA certificate passthrough for NixOS: https://github.com/dagger/dagger/pull/13137. I stumbled on this and found a TODO in the codebase so I went ahead and tackled it. Feel free to ping me if you need anything or have questions.

GitHub

GitHub is where people build software. More than 150 million people use GitHub to discover, fork, and contribute to over 420 million projects.

tepid nova
still garnet
civic yacht
#

FYI bunch of networking errors in CI right now from go module proxy, e.g. engine/client/drivers/container.go:17:2: github.com/google/go-containerregistry@v0.21.4: read "https://proxy.golang.org/github.com/google/go-containerregistry/@v/v0.21.4.zip": stream error: stream ID 1; INTERNAL_ERROR; received from peer

This has happened in the past too when proxy.golang.org was having issues, they don't have a status page that I know of unfortunately.

tepid nova
wild zephyr
#

discord discussion about a user looking forward to implement a --fail-on-cache-miss flag to better understand if their pipelines are consistent cc @civic yacht

civic yacht
#

Maven seems to be mad at us today, lots of Java tests failing in CI randomly due to 429s from maven

tepid nova
#

πŸ™‹ does anyone know why apk fails to work in our engine-dev playground container?

#
dagger call -m github.com/dagger/dagger engine-dev playground with-exec apk,add,openssh
leaden glade
#

@still garnet are self calls available in Dang? I guess not, but just wanted to be sure (and vote to add them πŸ˜‡ )

#

My use case: I'm changing the way SDKs are working, especially I'm removing the moduleTypes (going back to the previous version with entrypoint and empty function name for various reasons).
With that, codegen phase is in fact three things:

  1. analysis of the module source code (extract types and functions exposed): this requires the introspection json
  2. generation of the entrypoint (equivalent of the moduleTypes and the switch to invoke the right function): this requires the result from 1.
  3. generation of the dep bindings: this requires the introspection json, and 1. if we think about self calls
    My main idea is to have all that under one single call from the engine perspective (because it simplifies things). But for caching purpose I'd like to be able to call the different phases as separated calls so they can be cached.
    And the way to have that would be the use self calls so each function is cached individually instead of the single one. So better performances.
leaden glade
# tepid nova πŸ™‹ does anyone know why `apk` fails to work in our engine-dev playground contain...

It's because we are using wolfi as the base. And even if we are providing the apk-tools package, there's no available package database or repository configuration.
I've added a new arg to install packages:

dagger call -m github.com/eunomie/dagger@playground-extra-packages engine-dev playground --extra-packages openssh terminal

https://github.com/dagger/dagger/pull/13182

GitHub

Allow to install Wolfi packages in the playground
$ dagger call engine-dev playground --extra-packages openssh

charred lotus
tepid nova
tidal spire
#

It's because we are using wolfi as the base

Really its because we're using our wolfi module as the base, though, and we could change that module. Wolfi itself (cgr.dev/chainguard/wolfi-base:latest) does have apk and package repos

tepid nova
tepid nova
#

Aaaah. I thought it was some special wolfi thing I didn't know about

#

But what's special about our base?

#

I would have assumed it's just the regular base, since it's supposed to be a generic wolfi module?

tidal spire
#

afaik its the way we build the rootfs. Our wolfi module wraps our alpine module which uses our apko module to build the packages and add them to the fs

#

and its not actually the wolfi container, its just a container which we've added binaries to from the wolfi package registry

#

i guess you could argue what is a wolfi container, but in our case afaik it is not cgr.dev/chainguard/wolfi-base, but rather a scratch container with packages added

charred lotus
civic yacht
tidal spire
tepid nova
#

I think I may even have added that --base argument specifically because of this problem, and forgotten πŸ˜›

#

My full command now:

dagger call -m github.com/dagger/dagger \
 engine-dev \
 playground --base=alpine \
 with-unix-socket --path=/tmp/ssh.sock --source=$SSH_AUTH_SOCK \
 with-env-variable --name=SSH_AUTH_SOCK --value=/tmp/ssh.sock \
 terminal

Boom playground with ssh and ssh agent forwarding out of the box πŸ™‚ Now I can actually develop and push code straight from the playground.

Now just add some sort of persistent workspace concept, support for it in dagger cloud, and we've got ourselves one of those fashionable sandbox hosting services πŸ™‚

charred lotus
#

Quick question: I am changing all the non-digest pulls to be pull by digests, at least in our tests. Should I change modules/alpine's to be pull by digest or that's not a good idea?

tidal spire
tepid nova
#

Yes. In main you can already use that feature and delete all hardcoded digests from module code

#

(but not in 0.20.8 I believe)

charred lotus
#

ok i'll try that

#

(we're trying to fix the 429s in CI)

tepid nova
#

Ah that would be sweet!

#

You need to run with dagger --lock=pinned in CI, to disable looking up tags that are already in the lockfile

#

and --lock=live locally I believe, in main lockfile is disabled by default

tepid nova
# tepid nova My full command now: ```console dagger call -m github.com/dagger/dagger \ engi...
dagger call -m github.com/dagger/dagger \
 engine-dev \
 playground --base=alpine \
 with-exec --args=apk,add,openssh \
 with-unix-socket --path=/tmp/ssh.sock --source=$SSH_AUTH_SOCK \
 with-env-variable --name=SSH_AUTH_SOCK --value=/tmp/ssh.sock \
 with-exec --args=ssh,-T,-o,StrictHostKeyChecking=no,git@github.com \
 stdout
Hi shykes! You've successfully authenticated, but GitHub does not provide shell access.
obsidian rover
#

Is the golang:check failure known, related to Helm ?

tepid nova
civic yacht
# obsidian rover Is the `golang:check` failure known, related to `Helm` ?

I looked at that briefly, it was something about not being able to re-use the same name for something. I suspect the problem is we were starting to actually get caching on it πŸ˜„ It probably relied on never being cached accidentally. I bet there's just some state persisted in a cache mount that is resulting in us trying to re-use a name that already exists in that state

#

That error should go away for now again since I flipped the infra cache mount off again temporarily, but we can fix it once that's back on

civic yacht
#

@leaden glade I saw that we have code that does a cloud engine scale out for generators: https://github.com/dagger/dagger/blob/909a48fb2c58c8ac40c3699bad459af6205d36c8/core/modtree.go#L518

Which I'm confused by because scale out can not yet support transferring directories or anything besides scalars, so shouldn't work in theory. In practice when I just tried dagger generate --scale-out on my machine it error'd out with confusing errors.

You good to just delete it? I ran into this because I'm fixing a bug with how ModTree gets its cache serialized to disk, but then ran into a wall with tryRunGeneratorScaleOut since the code would essentially have needed to serialize state that is on a remote engine.

obsidian rover
#

I think we also have some flakes on moduleRuntime Python

obsidian rover
civic yacht
# obsidian rover > confirming rn by rerunning wihtout the cache volume + clean state, but i have ...

https://dagger.cloud/dagger/checks/github.com/dagger/dagger@cf18c4984897a843e88a9ca445aebab58fe7c190?check=test-split:test-module-runtimes&listen=268439c011162822&listen=4bea85e85d66058d&run=b2c5c43d-da82-4ae8-915a-dcd332114fde&test=TestPython/TestVersion/relaxed_.python-version&viewMode=tests#268439c011162822:L155

dagger
31  : ┆ ┆ ┆ ┆ ┆ Container.withExec ERROR [0.6s]
dagger
31  : ┆ ┆ ┆ ┆ ┆ [0.6s] | Resolved 39 packages in 114ms
dagger
31  : ┆ ┆ ┆ ┆ ┆ [0.6s] | error: Distribution `yarl==1.24.0 @ registry+https://pypi.org/simple` can't be installed because it doesn't have a source distribution or wheel for the current platform
dagger
31  : ┆ ┆ ┆ ┆ ┆ [0.6s] | 
dagger
31  : ┆ ┆ ┆ ┆ ┆ [0.6s] | hint: You're using CPython 3.11 (`cp311`), but `yarl` (v1.24.0) only has wheels with the following Python ABI tag: `cp310`
#

seems like an external dependency problem

#

googling around seems like similar things have happened to other packages when there was a bad publish to pypi

obsidian rover
civic yacht
charred lotus
#

And at the same time i'm trying to get to do --lock=live via scaleout ... scaleout has a lot of issues (possibly related to toolchains/workspace)

#
Error: check "helm:assert-template" not found in module "helm"

(with --scale-out only)

obsidian rover
charred lotus
obsidian rover
charred lotus
civic yacht
#

Anyone else seeing `TestEnvFile/

tepid nova
#

🧡 Checks as "rails" 🧡

obsidian rover
still garnet
#

dang self calls

civic yacht
obsidian rover
charred lotus
#

Not sure if it's related or not, but i also see a bunch of errors with --scale-out, and i think it's just due to the prefix dance logic. For instance check helm:lint looks for a helm:lint check inside helm, instead of looking for a lint check inside helm. Possibly generators have the same issue

civic yacht
civic yacht
charred lotus
leaden glade
charred lotus
#

FYI im fixing the check generator in main

#

Ah i didnt see you fixed it already thanks yves

charred lotus
wild zephyr
#

checking the golang:check failres in main

#

it's related to some k3s stuff

wild zephyr
tepid nova
#

<@&1506565370385793125> regression in workspace branch

tepid nova
civic yacht
#

Couple of fixes for some bugs when persisting/loading cache state between engine stop/start cycles: https://github.com/dagger/dagger/pull/13173

The low disk space problem we hit with cache volumes enabled in CI was because we were hitting those bugs, then the engine correctly fell back to "invalid state, throwing cache away" but then didn't delete the on-disk state for that cache being discarded, so it leaked. That's also fixed there too, it correctly deletes the state in that case.

charred lotus
#

any reason why TestTelemetry/TestGolden is not run in parallel? I see almost no CPU usage

wild zephyr
civic yacht
charred lotus
wild zephyr
#

I think we cal close your PR

charred lotus
tepid nova
#

PR cleanup @icy cliff

civic yacht
charred lotus
tepid nova
#

1.0 docs

dull crater
leaden glade
charred lotus
#

<@&946480760016207902> we're about to cut v0.21.0 (two PRs left). Any other issues/PRs we absolutely need to block the release on that you're aware of ?

civic yacht
charred lotus
tepid nova
#

1.0 dogfood 🧡

charred lotus
tepid nova
#

<@&1506565370385793125> prototype a change to the CLI progress renderers:

  • I'm assuming from memory that the default non -TUI renderer is --progress=main . if I'm misremembering just substitute the correct name in the rest of my message

  • Move the current "plain" to "classic". No longer the default

  • Change "plain" to be the most usable possible format for LLMs. take inspiration from all other renderers: logs, report, dots.

known issues today:

  • too much verbosity and internal details by default, "like stracing make". Especially for every day use ("which checks passed and how do I fix it?") as opposed to deep dive mode "what happened with this build, why was it so slow?" which doesn't belong in the default renderer

  • pipelines can run for a long time. some more concise renderers will not stream any progress for minutes- leaving llms wondering if they are hanging

  • missing contextual information that is available to humans in tui: how much time elapsed; cpu/memory/disk pressure; filesync is happening; (spitballing)

tepid nova
#

1.0 migration 🧡

fresh harbor