#maintainers

1 messages · Page 11 of 1

tepid nova
#

This could be generalized to every SDK, by setting some general rules:

  • A Dagger SDK should 1) write a client library, and 2) load a server library
  • Both libraries should live at a well known location
  • Both libraries should be convenient to use in either a standalone project or embedded in an existing project
#

If we could somehow make that work in all SDKs, then we could always default source to . in all cases, it would simplify a lot of things

civic yacht
#

If you want to extend the Dagger API with Functions, you develop that in ./internal/dagger/main. The Go SDK can generate boilerplate for you there.
I get the idea but then we are essentially changing the source dir path to be ./internal/dagger/main/ which I hesitate on given all the feedback we got on defaulting to dagger/ 😄

tepid nova
civic yacht
tepid nova
#

Where maybe we take it further than "the client bindings are an importable library", and also add "the module's implementation is also an importable (at least locally) library"

#

So in the case of the Go SDK, the generated main function would import "internal/dagger/main" or import "internal/dagger/entrypoint" or whatever we decide the package name is

#

In Python it might be src/dagger/entrypoint alongside src/dagger/sdk

civic yacht
#

Otherwise once we do add self bindings, it's going to be very confusing. On top of the fact that trying to get the author's implementation working in both of those modes is likely going to be very complicated and impose a bunch of requirements that don't seem worth supporting

tepid nova
# civic yacht "the module's implementation is also an importable (at least locally) library" -...

Yeah but I want that for a different reason. For project layout reasons.

The goal is not for me as a developer to import my own module implementation (technically I could still do that today by importing github.com/dagger/dagger/dev) it's for the SDK to have a place to store these files that doesn't conflict with an existing Go/Python/Typescript project. Which in turn solves our unresolved "source path, dev module?" dilemma

#

The only code importing that sub-package will be the generated runtime's main function

#

So you want bindings to be a library for a reason, and I want the server-side implementation to be a library for another reason, what if we made them both libraries, and standardized where they both live, and make that the new standard DX, to solve a bunch of problems all at once

#

Each SDK is free to decide where each library lives, to respect native conventions, but they have to be libraries (as opposed to full-blown "apps" like today)

civic yacht
# tepid nova Yeah but I want that for a different reason. For project layout reasons. The go...

Oh I thought you meant it would be importable and usable as a library by non-dagger code. As long as we aren't trying to support that, SGTM. I still find user feedback on the paths where they author module code hard to predict, so am still cautious in a non-blocking way, but I get the theory and agree that makes sense.

There's a bunch of corner cases around internal packages, having separate go.mods, etc. that we'd need to figure out with the dependency codegen bindings, etc. But agree on a high level. Have to go do the release so TBC on an issue 😄

tepid nova
#

In summary, this possible approach would be:

  • dagger init and dagger develop always set "source": "." by default.
  • Changing source would be considered an escape hatch, and relatively rare (unlike today)
  • Each SDK would be responsible for generating client bindings in a well-documented, static location within the module source. The location should be the most standard possible location for a sub-package of a project. The SDK is encouraged to make sure that sub-package can also be imported by an existing project.
  • Each SDK would also be responsible for 1) loading a module implementation from another sub-package, to be developed by the module author, and 2) generating a boilerplate for said sub-package, to help the author get started.
  • SDKs should not assume the module is a standalone "app" - eg. every change they make to the module source directory (creating, deleting or modifying files) should assume there is another "tenant" in the source directory, and minimize disruption
#

@civic yacht in the case of Go, are you saying that having generated client bindings in eg. internal/dagger is not enough today to proeprly use modules from an external client, because there also needs to be a generated dagger.gen.go in the caller package?

If that's the case, I'm OK with the location of those caller packages being a config field for the Go SDK in the dagger.json. "Hey in addition to internal/dagger/entrypoint, I would like to call the Dagger API from ./cmd/mytool and ./pkg/mything/plugins/dagger please and thank you 🙏 "

{
 "source": ".",
 "sdk": {
    "name": "go",
    "config": {
      "bindings_callers": ["./cmd/mytool", "./pkg/mything/plugins/dagger"]
    }
  }
}
civic yacht
# tepid nova <@949034677610643507> in the case of Go, are you saying that having generated cl...

I'm just thinking about cases where the module ends up with it's own go.mod but then a separate consumer of it (importing it as a library) wants to use it. Go doesn't let you import from under internal/ across go modules, so they would not be able to use it as a library. The fix could also be as simple as not putting it under internal/ in theory. But that has it's own implications worth thinking through, etc.

Making every path configurable in dagger.json does seem like the right approach provided we have good defaults so users who don't want to think about this don't have to (obviously I suppose, but important)

tepid nova
civic yacht
tepid nova
#

I'm OK with saying "if you want to import generated dagger bindings from your go module, add a dagger.json to your go module, with the dependencies you want bindings for"

tepid nova
#

@Erik Sipsma in the case of Go, are you saying that having generated client bindings in eg. internal/dagger is not enough today to proeprly use modules from an external client, because there also needs to be a generated dagger.gen.go in the caller package?

Still curious about this. I guess you need a local dagger.gen.go for dag. ?

civic yacht
tepid nova
# civic yacht Oh, devil in the details, but on a high level I think we can make this work. Whe...

Roger that.

Note that, in this model, worst case if it worked exactly as today (eg. caller needs its own dagger.gen.go) it wouldn't be the end of the world, it would just require an SDK config field in the dagger.json. Not quite as magical as everything "just working" without any custom dagger.json config. But still much much better than the current situation where you just can't use modules from an external client, period

#

@civic yacht another corollary to this approach: since we no longer would rely on "dev modules" (you just dagger init at the root of your project), we would need an alternative for testing modules. I think this naturally pushes us towards your idea of allowing a module to "test itself", a-la tes_xxx.go. My thinking is that there could be a special // +dev pragma to mark a function as "dev only", then a dagger --dev flag for loading your module in "dev mode". If that seems roughly directionally correct, I'll write up an issue to get the conversation going.

civic yacht
tepid nova
#

I don't know if it's related to the "otel draining" issues I keep hearing about, but on my "slow but not terrible" internet, there is a noticeable lag time (2-5 seconds) after my dagger commands complete, where it just sits there hanging before returning to the shell. Anecdotically, the more spans in the call, the longer it seems to take. So my guess is that it's uploading telemetry. Very noticeable, not sure if there's a solution but thought I'd share.

#

This is on very slow countryside broadband. Probably not noticeable on very fast home internet. But at this speed, Dagger is otherwise usable. So the 5s lag stands out

Screenshot_2024-07-18_at_11.07.51_PM.png
still garnet
#

It's kinda nice only having to configure that client-side

tepid nova
#

A future optimization perhaps - buffer otel in a local sqlite file, and start a background draining thread every time the CLI is called?

still garnet
#

I guess the CLI could forward its config over too 🤔

tepid nova
#

depending on the engine also means no telemetry on anything the CLI might do when an engine is not around

#

(which may or may not matter)

wet mason
#

No way to support maps in function signatures, right?

tepid nova
#

graphql doesn't have maps sadly

civic yacht
wet mason
#

@civic yacht Yep -- the widespread pattern (what we use in Cloud for e.g. map of span attributes) is a "custom" scalar Map (not so custom -- gqlgen for instance does it by default). https://gqlgen.com/reference/scalars/#map

Under the hood I believe it's just a JSON representation

scalar in graphql leaves stuff "raw" (but strongly typed) and then the marshailing/unmarshaling is left to the client and server

#

But yeah, not urgent

obsidian rover
#

Hey Erik, can I request some of your brain / bandwidth following the changes on Sockets 🙏

civic yacht
civic yacht
#

Actually I have a guess based on reading the error output and logs for 7979, sent an attempt out here: https://github.com/dagger/dagger/pull/7989 but need confirmation from @still garnet it's actually just handling an expected case in the tests vs. just side-stepping a bug without fixing it

civic yacht
fresh harbor
#

Do we have any command to generate dagger.gen.go (and friends) to all modules under the dev module? I could do traverse directory to do that. But I think you all might face this problems already. 😂

civic yacht
fresh harbor
#

Ah thank you. 🙏

tepid nova
tepid nova
still garnet
#

FrogeAlarm Probably need to ship a patch for this: https://github.com/dagger/dagger/pull/8001

tl;dr: the root span emitted by the dagger CLI never fails, so checks and traces never fail. The command itself still exits nonzero, so actual CI jobs will do the right thing; it's only a telemetry issue.

I have to head out for an errand but I can do the release for this when I'm back. Been a while since I've done one.

GitHub
fix root span never failing by vito · Pull Request #8001 · dagger/d...

A silly var capture issue. 😬
If ever there were a time for a linter to help...
Fortunately the command itself still fails, so everyone's CI should be doing the right thing, but traces in Cloud ...

civic yacht
#

fix root span never failing by vito · Pu...

still garnet
#

v0.12.2 release - 22nd July 2024

meager summit
#

hello folks, if I want to test changes to codegen locally, how could I do that

civic yacht
#

We also have tons of integ tests for various parts of SDK module codegen under ./core/integration/module*_test.go if that's easier to test something specific (i.e. test something with a specific module, etc.).

You can run individual tests there with dagger -m dev call --source=.:default test custom '--run=TestModule/<TestName>' --pkg=./core/integration

meager summit
#

but when I try running tests using: dagger call -m dev --source=.:default test important, it almost seems like it never ran that test

civic yacht
# meager summit but when I try running tests using: `dagger call -m dev --source=.:default test ...

Oh when you run test important it's running hundreds of test cases. And depending on your machine it may take quite a while so you might need to append an extra timeout. It takes 30m for me locally (8 cpus) so I usually do dagger call -m dev --source=.:default test important --timeout=40m to be safe.

We need to improve overall performance so they run faster locally, but in the mean time you should feel free to just run a smaller set of test cases locally. So in your case you could just do
dagger -m dev call --source=.:default test custom '--run=TestModule/TestGoInit' --pkg=./core/integration

meager summit
#

FWIW, yesterday when I ran the tests, it was taking forever (I am on Macbook M2 pro) and I stepped out. when I came back after 2hours, it was still running (probably stuck). 🙂

civic yacht
meager summit
#

what i am doing right now is:

  • i have a fresh copy of dagger checked out
  • verified latest version of dagger is installed (v0.12.2)
  • no dagger engine is running (verified using docker ps)
  • commented out everything inside TestGoInit function
  • added just one testcase
    t.Run("from scratch", func(ctx context.Context, t *testctx.T) {
        require.Equal(t, "a", "a")
    })
  • ran test using dagger call -m dev --source=.:default test custom --pkg="./core/integration" --run="^TestGoInit" --progress plain
  • it seems to be running much more than that specific testcase
#

I am wondering if somehow its using a cached copy of testcases somewhere?

civic yacht
# meager summit what i am doing right now is: - i have a fresh copy of dagger checked out - ve...

the syntax would need to specifically be --run=TestModule/TestGoInit (just double checking in case that was a typo).

When I run that I do get a bunch of output first for building the engine, but then it runs just TestGoInit test cases, which takes a total of like 3mins for me. We need to build the engine first to run the tests since they are all running against an engine built from your local code. Is it possible you were seeing the output of building the engine first? In --progress=plain mode specifically that will all kind of fly by and might look like tests cases running (it's a bit more clear when you don't use --progress=plain, the TUI output has been cleaned up a bunch in recent versions)

meager summit
#

yep, that was it. it had to be --run=TestModule/TestGoInit

#

thank you for the pointers and additional info

civic yacht
#

No problem at all, thanks for working on that issue!

civic yacht
#

✨ v0.12.3 - July 30th 2024?

spark cedar
#

👋 fyi y'all - i've just merged https://github.com/dagger/dagger/pull/7973
the idea is to try and encourage longer and more informative changelogs - please write down as much as you want, the person doing the release will edit/format/etc if it needs to be pared down 🙂

timber latch
#

Time to practice writing: For the night is dark and full of terrors...

civic yacht
#

Re-lobbying for reviews on https://github.com/dagger/dagger/pull/7943 Just to trim a chunk of the annoying test flakes we're hitting currently

Wait, I had to refresh the page apparently, it has been reviewed, thank you Jed 😄

obsidian rover
#

I'm hitting the path limit length with t.tmpdir on unix: 108bytes. What's your go-to strategy to shorten it usually ? I'm about to Sha it, but open to better ideas with better readibility 👼

meager summit
#

hello folks, I am still looking at https://github.com/dagger/dagger/issues/7941, and identified a potential issue on code-generation side (still trying to verify by running the e2e testcase).

Now code generation seems ok, but when I make a call to "dagger query", it fails with here: https://github.com/dagger/dagger/blob/4c58dd459a8e5e32d64356ec1805ede988a85c6f/dagql/objects.go#L193 with error: query has no such field: aModule". (my module name is a-module).

my question is does the engine load the schema:

  • from already generated code, or
  • from cache it possibly built during dagger --init, or
  • generates the code again on engine side and load it from there
  • or some other way
civic yacht
#

I'm hitting the path limit length with t

civic yacht
#

hello folks, I am still looking at https

spark cedar
#

I might also raise during our little architecture discussion later

meager summit
#

when running integration tests, is there a way to see engine logs. I am trying to trace a function call by using panic, but I don't see stack trace. only that engine returned 502 error.

still garnet
#

engine logs in integration tests

wet mason
#

@civic yacht 👋 Do you think "module self invocation" is still relevant?

civic yacht
wet mason
#

Ok great, that's what I thought, but it was from months ago wondering if that changed

#

AFAIK there's no issue tracking that (?), I'll create one

civic yacht
wet mason
#

Feel free to edit if I missed something

rancid turret
meager summit
#

hey folks, re: https://github.com/dagger/dagger/issues/7941, this issue is happening because of how strcase works for different inputs. specifically ToCamel function.

one of the fix could be to use our own version of strcase (or a completely different library), and verify it translates the strings as expected (and add testcases to verify/document those), but I am worried if changing that can trigger backward compatibility issues for existing dagger modules.

any thoughts or preferences?

wild zephyr
#

👋 planning to tackle this one https://github.com/dagger/dagger/issues/7527 quickly today since it's hurting some users https://discord.com/channels/707636530424053791/1265736714685513909. Sharing for visiblity cc @rancid turret @civic yacht

GitHub
Don't reuse parent go.mod by default · Issue #7527 · dagger/dagger

Problem When initializing a new dagger module with the Go SDK, by default no go.mod is created if a parent go.mod already exists. This is not the correct default behavior. It causes confusion, and ...

rancid turret
#

Docker build secrets

tepid nova
tepid nova
civic yacht
spark cedar
#

Left some quick feedback on a pass through, but happy to circle back round for more detailed comments/etc tomorrow 🙂

wild zephyr
#

Can you make sure there's a flag to opt-

hasty basin
#

@rancid turret
Was it intentional that WithExec() ignores the conatiner's entrypoint by default while AsService()does not?

#

I was planning to create an issue to harmonize the two if they should be the same

rancid turret
# hasty basin <@768585883120173076> Was it intentional that `WithExec()` ignores the conatiner...

Yes, it's intentional, but it's possible there's some details we overlooked for services. Context: https://github.com/dagger/dagger/issues/7821#issuecomment-2213999367

GitHub
CLI: don't run a container's default args when printing it · Issue ...

Context Since #6515, the CLI prints objects returned by a pipeline. To do this, it calls certain functions in the returned object, specifically functions which 1) return a printable value, and 2) h...

hasty basin
rancid turret
hasty basin
#

You have to add with-exec --args "" --use-entrypoint stdout
or at least with-exec --args "" stdout to get the defaultArgs to be used (if you don't need entrypoint)

rancid turret
#

Yes, that’s what I said. Falling back to the default command was removed from stdout but not asService. That was a separate breaking change from not using the entrypoint in WithExecs. The reasoning for not touching asService was that it seems likely when running services that you’d want to run CMD. But I agree that using the entrypoint there makes it inconsistent with when you do define a WithExec.

wild zephyr
#

@civic yacht @spark cedar do you know by any chance if the runtime containers make use of the proxy variables set in the engine?

I'm starting an engine with HTTP_PROXY=http://foo.com but dagger init --sdk go works without issues which makes me think that it might not be using the supplied proxy as I was expecting it to fail given the wrong proxy value

wild zephyr
stuck bloom
#

Can't speak for incorrect proxy but I have tested a correct proxy to work with Go etc.

wild zephyr
spark cedar
#

🧵 for discussing the upcoming v0.12.4 release 🎉

stuck bloom
wild zephyr
#

are you sure the proxy is being used Nipuna? 😅

wild zephyr
# wild zephyr yes. That's strange, since `go get` completely ignores the `HTTP_PROXY` variable... 
130|marcos:tmp/lala (⎈ |eks-prod)$ HTTP_PROXY=127.0.0.1:8080 http_proxy=127.0.0.1:8080 go get -u -v  github.com/fatih/color
go: downloading github.com/fatih/color v1.17.0
go: downloading golang.org/x/sys v0.18.0
go: downloading github.com/mattn/go-isatty v0.0.20
go: downloading github.com/mattn/go-colorable v0.1.13
go: downloading golang.org/x/sys v0.22.0
go: trying upgrade to golang.org/x/sys@v0.22.0
go: added github.com/fatih/color v1.17.0
go: added github.com/mattn/go-colorable v0.1.13
go: added github.com/mattn/go-isatty v0.0.20
go: added golang.org/x/sys v0.22.0

^ this just works and I was expecting it to fail

civic yacht
wet mason
#

@civic yacht 👋 How unlikely it is for us to be able to support bind mounts from the host? I'm going with the assumption of "very unlikely", just wanted your thoughts before I explore alternatives further

civic yacht
# wet mason <@949034677610643507> 👋 How unlikely it is for us to be able to support bind mo...

It depends on what you mean. If you mean "bind mount from the client's host into the container", then no it's not possible because we don't assume the client and the engine are on the same host. If you mean "bind mount from the engine's host into containers", then it's technically possible, but not sure we want that as it's very confusing; it's the equivalent of connecting to a remote docker with DOCKER_HOST, running docker run -v /foo:/bar ... and then getting very confused when the bind mount you end up with is not from your host

#

There are options to create the same sort of effect as a bind mount through continuous filesync (e.g. mutagen or similar), but not technically a bind mount

wild zephyr
civic yacht
wet mason
#

upside is it works remotely

civic yacht
# wet mason Yep, that's what I'm looking into ... wanted to get your opinion on volumes befo...

Yeah filesync is very much in the Hard Problem ™️ category.

Storing cache volumes on the host is somewhat related to all this in that it lets you mount in a directory from the engine's host into containers. After a bunch of failed attempts the approach I've settled into is writing a containerd snapshotter implementation to support it, which is interesting because it makes it more reusable in other cases potentially. I've started thinking we may want to consider reconceptualizing the whole thing from "cache mounts" to more like "persistent volumes" (which cache mounts are a subset of)

#

Not sure what use cases you have in mind and if they would be related to all that potentially

wet mason
civic yacht
# wet mason the use case is giving a quick attempt to this (live dev of dagger cloud with ho...

Cool, yeah if you haven't already done this, mutagen supports different sync behaviors that have different performance. I think two-way read-write sync is the slowest, but for this case we probably just want one-way "client -> server"? Which could make it faster maybe.

Also, even if the initial sync is slow, provided that is cached and mutagen knows how to reuse existing directories for sync, it may be a one-time up front cost that's acceptable since it'll be faster later

#

I do think that it would be nice to generalize the whole volumes concept here, beyond just cache mounts. E.g. you could use the API to create a volume that is automatically setup with mutagen and synced with the client

wet mason
#

Oh interesting. Right now only attempted a very simple test using ssh transport, I was thinking we could embed the mutagen agent on the shim and the client on the CLI

#

However, you got me thinking ... maybe using a volume concept (starting with a cache volume -- assuming it can be shared) could be a beter option

#

Also right now I'm working with services ... not sure if "file sync" should be limited to services or broadly available

civic yacht
# wet mason Oh interesting. Right now only attempted a very simple test using ssh transport,...

Yeah for the server side I have imagined this as "Service running mutagen that has the volume mounted in; any other containers with that volume mounted in will receive all the updates". I like that because it's just re-using existing features and could be generalize beyond mutagen.

The client side is tricky though because yeah it's either "embed mutagen" (in which case it's not generalized anymore, hardcoded to mutagen) or "ask user to start mutagen locally", which is not nearly as seamless

wet mason
#

Oh ... also also -- we can only Up() services from the CLI to get h2c networking, right? e.g. if we Up() them from a module then it's actually "module to container" rather than h2c networking? Meaning right now we can only "up" a single service for h2c?

wet mason
civic yacht
#

It's dumb in this case since there's no "real" network dep between them, but that would work for now. But that's why this may all call for some new primitives. Like the engine should know "this container mounted in this volume, that volume is backed by this other service so I gotta start that service"

wet mason
#

sweet, nice workaround

wet mason
#

but I'm looking into workarounds, currently reading through the "mutagen compose" codebase, they use a different approach with a sidecar container, trying to see if we can re-use that

civic yacht
wet mason
#

@civic yacht The volume approach is very nice though, I'm going to try that

#

Initially I was looking into starting another process in the service, or re-using the shim

#

but the volume approach is much simpler and better

#

and caching

civic yacht
# civic yacht Yeah I'd be totally fine with embedding it for now, better to have the feature e...

This gets into huge separate cans of worms, but one way to generalize this support on the client side would be WASM and/or rootless containers. Then every part of this could be implemented by users. They write a "service-backed volume" as a module and can return a container for the client side that's run via wasm/rootless-support. Then anyone could swap out mutagen for something else, or potentially use this for completely different types of use cases.

Dreaming here obviously, but not implausible in the long term 🙂

wet mason
civic yacht
# wet mason Misunderstood this bit ... with a service binding both services come up, but I c...

Ah okay, that makes sense. So just to double check:

  1. The client wants to up the "code dev" service and the "mutagen" service
  2. The client also wants to proxy ports from both of those services locally

Right? If so, I think you could for now use https://daggerverse.dev/mod/github.com/kpenfound/dagger-modules/proxy@beaf4600637b3f492cd3e1e1598ef799d8cd3164 to group multiple services into a single one.

Which is a beautiful module and amazing it's possible, but really this should probably be something built-in to the core API

proxy :: Daggerverse

Proxy multiple services though a single service

wet mason
civic yacht
#

Yeah if you want to encapsulate this into a single "dagger call ... up" I think 1) or 3) would have to be how you do it. I like 1) more because it's more easily re-used without customizing each service.

  1. works but would require running multiple "dagger call ... up"s in parallel
wet mason
#

👍

#

Just tried 2 to give it a shot and it works 🙂

#

I'll look into cleaning that up

# 
func (m *Livedev) Dev(ctx context.Context) (*dagger.Service, error) {
    code := dag.CacheVolume("code")

    app := dag.Container().
        From("python").
        WithWorkdir("/code").
        WithMountedCache("/code", code, dagger.ContainerWithMountedCacheOpts{
            Sharing: dagger.Shared,
        }).
        WithExec([]string{"python", "-m", "http.server", "8080"}).
        WithExposedPort(8080).
        AsService()

    return app, nil
}
#

(that's the cloud frontend mutagen'd into the dev contianer)

wild zephyr
#

sharing a friendly one-liner I use to build all the packages in the dagger/dagger repo excluding the /docs folder as those fail if ./... is supplied. go list ./... | grep -v docs | xargs go build

no sure if there's an "easier" way? 🤔.

might be worth adding a dagger function for this maybe? cc Justin

twin crow
#

Hey, I am working on tracking few metrics from our Github issues (number of issue "kind/bug" created per week / month). Same for merged PR. I am going over our issues and PRs. Can we make sure from now on we properly label the PRs with kind/bug when it's a bugfix? That will help get some useful insights.

obsidian rover
#

Following a conversation with Andrea to ensure the SSH ref style vision / DX impact, your feedback is more than welcome as we approach the final stages of the private support ref style: https://github.com/dagger/dagger/issues/6113#issuecomment-2264712473 🙏

Current pending PR implementation will be tweaked tomorrow to fit this new last ref style

bronze hollow
#

I've not had any pings from GitHub about the contextual module issues for a while now. Are these still in progress?

wild zephyr
bronze hollow
#

That's awesome. Can't wait!

wild zephyr
fair ermine
rancid turret
#

@fair ermine, it appears the Python code in the PR fixes an issue that I wasn't even aware. It changes a required arg to an optional one (correctly), but that means it also changes the signature when it's being used (in /dev). Basically you get a function arg with a default but no optional(True). I think the engine should enforce it. The problem with the root /dev is it needs to run both on a released engine and a dev one. So we need the compat layer here. I'll work on fixing it in a separate PR.

fair ermine
rancid turret
#

Actually, previous behavior makes sense, in that having a default doesn't necessarily mean it accepts a None (null), which is what withOptional(true) means. This has to do with https://github.com/dagger/dagger/issues/6749.

fair ermine
rancid turret
# fair ermine In TS, if there's a default, I set it as no-optional, except if there's an objec...

Problem is you lose the expressiveness in GraphQL, that's why I needed a bit of time to investigate more on that issue. What I mean by expressiveness is that you can have non-nullable fields with default values. For example:

name: String! = "main"

If you set withOptional(true) whenever there's a default value you won't be able to get the result above. We do have a test case that suposedly covers that, so I'd need to investigate what you're saying.

For now you don't see any real effect on making it nullable, but as I show in the issue, the API is ignoring null values (considering them omitted). I've reached out to the core devs a while ago and it it's not intentional. When that's fixed and you force a withOptional(true), you'll possibly get a null on a param that doesn't allow for it. So you need to buffer for that case in the SDK.

#

Still... it's important to make things intuitive in the SDK side, but rather than rely on IS_NULL in codegen, it should also consider default values before marking an arg as optional or required.

fair ermine
#

Okay!

rancid turret
# fair ermine Okay!

Hey, you can merge and rebase this:

GitHub
chore(ci): Adjust dev module incorrectly detecting optional argumen...

This is a workaroud until the following is fixed:

#8106

Not a part of that PR because it's needed for:

#7744

GitHub
fix(sdk/go): consider default values when grouping optional argumen...

Part of #6749
Fixes #6964

WarningThis is a breaking change for Go SDK users when depending on a module that has a non-null function argument with a default value. With this change, the argument wi...

spark cedar
#

👀 turns out, you can put Container.From in the middle of a chain... it doesn't need to be the first item in it.
is this what we want? could we potentially break this in the next release?
(definitely didn't spend a few minutes tracking down where my nice container went)

rancid turret
#

But only ImageConfig stuff are merged. Mostly lists and maps. The rest is overwritten.

spark cedar
#

i get the logic for keeping Env - but it's really weird for the root filesystem to get silently overwritten

rancid turret
spark cedar
#

i get the logic for merging the image config + similar - i just don't think we should rewrite the FS if one has already been created

#

e.g. doing .From.From should be invalid (but doing .WithEnv.From seems fine)

#

or in my case, i accidentally did dag.Wolfi.From and was surprised when dag.Wolfi just disappeared

wild zephyr
#

👋 just a heads up that @meager summit will be helping out with various OSS engine issues in the upcoming weeks. He's taking https://github.com/dagger/dagger/issues/7941 as a starter and will very likely need our input to streamline his understanding of the codebase 🙏

GitHub
🖐️ Quickstart issues with Go (`dagger init` + `dagger call`) · Issu...

What happened? What did you expect to happen? Hey! I'm testing out Dagger for the first time and followed the quickstart guide. When I ran dagger init --sdk=go in the root of my existing reposi...

wild zephyr
#

👋 helping a user here (https://discord.com/channels/707636530424053791/1270424034659799051) I've realized that WithMountedCache doesn't respect the currently set container user to set the Owner flag which might cause some non-desired behaviors as seen on the help request. It'd be nice if this could be handled automatically. @civic yacht any preliminary thoughts before opening an issue to address it?

civic yacht
# wild zephyr 👋 helping a user here (https://discord.com/channels/707636530424053791/12704240...

Using the container's user feels reasonable to me, but the current implementation isn't great in that the owner field only works the first time a cache volume with a given key is used (mentioned in the docs for the API: https://github.com/sipsma/dagger/blob/479c8251479b993ca1d703e138310a26665cbe49/docs/docs-graphql/schema.graphqls#L600-L600)

I've been thinking a lot about this since I need to support it in the new cache volume implementation obviously. It's surprisingly kind of tricky. The only "good" solution I know of is to use id-mapped mounts, but that's a pretty recent linux feature so we can't rely on its existence yet: https://man7.org/linux/man-pages/man2/mount_setattr.2.html

The solution I'm going with for now is that we will always honor the owner setting of the mount (different than current implementation) but for now you will get different cache volume instances for each owner setting. This still is not ideal, but I prefer it since you at least always get the owner you asked for

wild zephyr
#

Using the container's user feels reasonable to me, but the current implementation isn't great in that the owner field only works the first time a cache volume with a given key is used (mentioned in the docs for the API: https://github.com/sipsma/dagger/blob/479c8251479b993ca1d703e138310a26665cbe49/docs/docs-graphql/schema.graphqls#L600-L600)

That's fine and it makes sense it works this way until better underlying features like id-mapped mounts / shiftfs become more popular. Still I think that using the Owner field the first time the volume is created is a progress to what we have today

#

The solution I'm going with for now is that we will always honor the owner setting of the mount (different than current implementation) but for now you will get different cache volume instances for each owner setting.

Oh I see.. so you'll be changing the behavior of how ownership work in cache mounts. Is your plan to attach the Owner attrbiute to the CacheVolume type now?

civic yacht
#

I was considering doing that and deprecating the setting on ContainerWithMountedCacheOpts, yeah. Same for the Source field there since it requires the same behavior of actually creating a new cache mount for each Souce value.

#

The current behavior is also extra weird because internally what happens now is buildkit mixes in the low-level cache ref ID of the Source, which is a completely random value associated with buildkit internals, so the "real" cache volume ID is unpredictable. Another part of the changes I'm making is for the cache volume's unique ID to be based on a content hash of the Source, so it's at least consistent and predictable in that sense.

wild zephyr
civic yacht
wild zephyr
fair ermine
tepid nova
#

So many conflicts in all my PRs...

tepid nova
#

Version management feedback: I'm on dagger 0.12.2. Getting this error: failed to load sdk module github.com/dagger/dagger/sdk/php: select: failed to create module: select: module requires newer engine version: version v0.12.2 does not meet required version v0.12.3

Isn't it excessive that a module requiring 0.12.3 fails to load with an engine 0.12.2?

bronze hollow
#

Thanks Dagger team for your support! I enjoyed that

wild zephyr
tepid nova
#

@still garnet @dense dust @fair ermine with context directory (almost) merged, is there an opportunity to use the context directory as a more reliable source of git metadata, for cloud?

dense dust
tepid nova
#

basically the workdir becomes irrelevant (IMO)

#

So all the edge cases might just melt away

dense dust
#

well that would be awesome

#

I'll take a look

tepid nova
#

We may need to add support for a --context flag for certain cases

#

@dense dust do you have a writeup of all known edge cases for git metadata? Situations where it doesn't work properly

dense dust
fair ermine
meager summit
#

hello folks, when running tests for PR

meager summit
spark cedar
#

👋 it's thread time for v0.12.5 👀

tepid nova
tepid nova
#

Question on our Github Actions configuration.

There is a lot of boilerplate config for allowing us to run a dev dagger engine. What is that for? Don't we already use dagger-in-dagger all over the place? Do we really need to also do this in YAML?

#

cc @astral zealot @stray heron

spark cedar
#

I think I agree on the "return object on sync" point, I quite like that - but IMO, that's a super breaking change, and it feels a little orthoganal? Maybe, I'm still tring to wrap my head around what it would mean

tepid nova
#

General comment on our GHA configuration: LESS YAML ENGINEERING PLEASE

#

The complexity is insane

spark cedar
#

it's so much better than it was 😄

#

but point taken

tepid nova
#

I'm trying to do passes of simplification after the fact, but it's a major time investment for me, it would be much easier if the focus on simplicity, and avoiding yaml engineering, were infused from the start

#

We're telling our users "Dagger will make your yaml smaller, and less weird and unique" and we're doing the exact opposite

rancid turret
#

Sync

spark cedar
rancid turret
spark cedar
#

Ahh good catch, my bad will fix that 🎉

tepid nova
#

I'm hitting a weird bug in Directory.terminal(). If the directory contains a withNewFile() then the terminal fails with "exit 1".

#

This works for me:

dagger call -m github.com/shykes/core directory terminal

But this doesn't:

dagger call -m github.com/shykes/core directory with-new-file --path=hello.txt --contents='hello world' terminal

On vanilla 0.12.4

cc @wet mason

rancid turret
#

This works too:

dagger core directory with-file --path=README.md --source=README.md terminal
tepid nova
#

@spark cedar @astral zealot @stray heron sadly I won't be able to attend the demo call later today, but I wanted to share my side project with you: a dagger module to generate Github Actions configurations 🙂 My goal is to replace our unique snowflake YAML with a clean generated config.

#

Example:

dagger call -m github.com/shykes/dagger2gha \
  --public-token FOOBAR \
  --dagger-version latest \
  on-push --branches=main --command '--source=.:default engine test important' \
  on-pull-request --command 'engine test all' \
  on-pull-request --command 'sdk python test' \
  on-pull-request --command 'sdk go test' \
  config \
  -o ./my-repo

Resulting file tree:

$ find my-repo
my-repo
my-repo/.github
my-repo/.github/workflows
my-repo/.github/workflows/push-1.yml
my-repo/.github/workflows/pr-1.yml
my-repo/.github/workflows/pr-2.yml
my-repo/.github/workflows/pr-3.yml

Each trigger is its own workflow, with a single job. Contents of pr-3.yml:

{
 "name": "sdk go test",
 "on": {
  "pull_request": {}
 },
 "jobs": {
  "dagger": {
   "runs-on": "ubuntu-latest",
   "steps": [
    {
     "name": "Checkout",
     "uses": "actions/checkout@v4"
    },
    {
     "name": "Call Dagger",
     "uses": "dagger/dagger-for-github@v6",
     "with": {
      "args": "sdk go test",
      "cloud-token": "FOOBAR",
      "module": ".",
      "version": "latest"
     }
    }
   ]
  }
 }
}
tepid nova
#

I think eventually this approach could make the dagger-for-github action unnecessary. Instead you just generate the yaml blog that you need. Instead of parametrizing the github action in yaml (complex and brittle), you parametrize the generating pipeline instead. Then the generated yaml can be much simpler, because it has no dynamic parameters.

#

btw @spark cedar it includes a complete (or almost complete) native GHA schema. If you want to experiment with exposing the entire low-level schema as a Dagger API, it should give you a solid starting point. But sadly the native types don't all translate directly to Dagger, because there are maps...

wild zephyr
tepid nova
wild zephyr
meager summit
#

hi folks, after pulling in the latest changes, the typescript sdk has these errors (on clean main branch). I tried running dagger develop inside runtime dir, but that didn't help.:

./main.go:36:16: undefined: Directory
./main.go:49:17: undefined: Directory
./main.go:65:71: undefined: ModuleSource
./main.go:65:104: undefined: File
./main.go:65:112: undefined: Container
./main.go:107:65: undefined: ModuleSource
./main.go:107:98: undefined: File
./main.go:107:106: undefined: GeneratedCode
./main.go:130:69: undefined: ModuleSource
./main.go:130:102: undefined: File
./main.go:130:102: too many errors

Also when I run the typescript based integration tests it works on main (but fails with ^^ error on my branch). I am using following command to run the tests:

any pointers what I could be doing wrong here?

#

only difference I can think of is that typescript-runtime/dagger.json is pointing to dagger v0.11.3, but with my branch I am using latest from main?

spark cedar
#

hm, the typescript runtime needs updating

meager summit
#

Yeah that is what i thought

spark cedar
#

if you're around that area, help updating the typescript sdk to use a v0.12 version would be super useful (otherwise, i can pick that up)

meager summit
#

I am happy to do it, let me check what that entails. Thank you for confirming.

#

I have one more question though, the CI integration tests should also fail due to this? Like it is failing for me on my local run?

spark cedar
#

with v0.11, the new dagger engine is attempting to emulate the old api - so it still works

#

but with v0.12 and more, it drops this emulation, and so breaks

#

you can run dagger develop without doing the engineVersion bump, by passing the --compat flag to dagger develop

#

that might help unblock you if you were working on something else and don't want to get distracted here

meager summit
#

Yeah, i would love to be able to finish the strcase lib replacement ticket first :), i dont want to jinx it but it seems like typescript tests are the only failing tests right now

#

But if i get around to it, i will post it here before starting to ensure no duplicate effort is done on that update

#

Thanks again for the pointers

spark cedar
#

awesome awesome, thanks 🙏

#

feel free to ping more aggressively once the strcase replacement pr is ready to go 😄 sorry, it's been so busy the last week

meager summit
#

Sure thing, thank you

tepid nova
#

Raising a red flag on possible blocker for "daggerizing without littering": do we actually know how a Typescript module can co-exist with an app in the same repo? https://github.com/dagger/dagger/issues/8123#issuecomment-2284536121

GitHub
Daggerizing without littering · Issue #8123 · dagger/dagger

Problem By default, daggerizing an existing project (either with dagger init --sdk=NAME, or dagger develop --sdk=NAME) will cause littering, by mixing Dagger-managed files with pre-existing files. ...

spark cedar
meager summit
#

btw, the strcase lib PR is ready for review/feedback. There is one pending question, but otherwise it looks ready for feedback

spark cedar
#

awesome, i'm doing a lot of reviews/etc today, so will get on it! 🎉

astral zealot
#

Currently hitting an issue while testing some ideas that require the use of dagger in dagger.

I have this container where I install the dagger CLI, clone a repository with a module and do a dagger call with priviliged nesting, this works just fine:

    return dag.Container().
        From("ubuntu").
        WithWorkdir("/home/ubuntu").
        WithExec([]string{"sh", "-c", "apt update && apt install -y git-core curl"}).
        WithExec([]string{"sh", "-c", "curl -LO https://dl.dagger.io/dagger/install.sh && chmod +x install.sh"}).
        WithExec([]string{"sh", "-c", fmt.Sprintf("DAGGER_VERSION=%s BIN_DIR=/usr/local/bin ./install.sh", daggerVersion)}).
        WithExec([]string{"sh", "-c", "git clone https://github.com/shykes/daggerverse"}).
        WithWorkdir("/home/ubuntu/daggerverse/hello").
        WithExec([]string{"sh", "-c", "dagger call hello"}, dagger.ContainerWithExecOpts{ExperimentalPrivilegedNesting: true})

However, when I run both the git clone and the dagger call in the same command:

        WithExec([]string{"sh", "-c", "git clone https://github.com/shykes/daggerverse && cd daggerverse && dagger call hello"}, dagger.ContainerWithExecOpts{ExperimentalPrivilegedNesting: true}).

It fails indicating that there is no module source available:

input: module.withSource.initialize resolve: module name and SDK must be set

I'm not entirely sure how buildkit works, but I imagine this has something to do with a command generating layers that dagger then tries to use but somehow have not yet been synced 👀 ? Not sure if there is a workaround to this. Tried with cache volumes to see if that would help but same issue

obsidian rover
astral zealot
obsidian rover
# astral zealot That was my first guess, but its the same result. It also would not explain why ...

Dug a bit

Current guess (which joins yours, but to be confirmed by the experts ahah) is that the setupNestedClient is being called before the actual run of the exec, when the ExperimentalPrivilegedNesting flag is set to true:

  1. we set the clientID to a non-null value when option is true: https://github.com/dagger/dagger/blob/main/core/container_exec.go#L122

  2. Prior running the step, some setupFunctions are run: https://github.com/dagger/dagger/blob/main/engine/buildkit/executor.go#L162

  3. setupNestedClient checks that value, which is then non-null and sets up the filesyncer:
    https://github.com/dagger/dagger/blob/main/engine/buildkit/executor_spec.go#L899-L903. This newly created session will thus have a state without the git repo

  4. Lastly, I believe that this setupNestedClient just prepares any nested dagger client to talk to this nested session

So to me, you are indeed copying the repo (I checked with an ls and some sleep) in the main session, and then the nested session state's doesn't resync after as it was done as a setupFunction.

**If correct, I have no solutions / wouldn't know how to help though, sorry. I'll let the big guys talk ahah 👼 **

wild zephyr
astral zealot
# obsidian rover *Dug a bit* Current guess (which joins yours, but to be confirmed by the expert...

Thanks for the deep dive!! party_gopher

That makes sense, what I'm not sure is what would the correct behavior be. We can probably argue both ways, what is a bit confusing is that in the filesystem of the underlying container where the commands are being executed the files are there. Like you mentioned, doing an ls shows that the contents exist, so its a bit counter intuitive that a dagger call wouldn't be able to pick them up. Not sure either if its possible to fix, I'll look further at the code and see if I can grasp a bit better the overall structure

wild zephyr
spark cedar
#

mm yeah, i think this is probably a bug... i am confused as to how this happens still. I think @obsidian rover's right, this probably is something to do with setting up the nested client
that said, i think the NewFilesyncer code is right - it's just passing the path, so when it gets populated later, it should be fine right?

#

if someone could put together an integration test, that would be super useful, either posting it as a comment in an issue, or opening a draft pr

#

i think this probably deserves some more time to properly investigate, i don't have any immediate ideas

wild zephyr
#

that said, i think the NewFilesyncer code is right - it's just passing the path, so when it gets populated later, it should be fine right?

but IIUC the Filesyncer won't see any new changes in the path after the WithExec has been started? So reproducing Matia's issue if you do touch foo.txt && dagger call in the same WithExec, the Filesyncer won't pick the change, correct?

spark cedar
#

wait

#

the command is wrong

#

git clone https://github.com/shykes/daggerverse && cd daggerverse && dagger call hello

#

the dagger.json at the root of shykes/daggerverse is invalid

#

you cannot call it

wild zephyr
#

@spark cedar you get the same error if you do cd daggerverse/hello

spark cedar
#

hm

#

you're right

#

tl;dr @obsidian rover is right - however, the state is 100% correct, it seems to be related to the current directory

#

the error message about missing the sdk and name is indicating that no matter what directory we cd into, we're getting the top-level (as the filesyncer code seems to kind of indicate might be happening)

astral zealot
#

@matipan seems like we forgot the

spark cedar
#

trying to decide whether we need to do module compat here (because if we do, it's going to be such an unenjoyable amount of tedious work, and i would love to avoid doing/reviewing it if at all possible)

#

and for sure, in principle we should definitely be adding it, but it's just such a huge expansion of module compat into every little corner of the codebase

wild zephyr
#

what's the comment particularly? the link doesn't take me to a specific one 😬

spark cedar
#

oops, updated the link

tepid nova
#

quick question on context dir. When the caller's module and callee module are not the same, which context does defaultPath? Caller or callee? I assumed caller because it's way more useful that way. But it never came up during implementation, so I am suddenly dreading that it's implemented as callee

tepid nova
#

I'm starting to wonder how a function might write to the context directory... This would make codegen use cases smoother. cc @still garnet I haven't forgotten about our dagger call fix conversation, I still want to do it... Would be nice if the command really could be that short. no -o PATH necessary...

Would love to hear thoughts on this!

tepid nova
fair ermine
meager summit
spark cedar
#

yup yup, we're on it in the release thread

tepid nova
fair ermine
tepid nova
tepid nova
#

@spark cedar @still garnet I started getting weird errors in my experimental CI pipelines, that seem to coincide with the 0.12.5 release. I reverted the pipeline to 0.12.4 (yes that is a supported option of my generator) but still getting weird errors. They are go build errors that seem to involve opentelemetry and bass somehow, so I thought I'd copy you @still garnet 😛

spark cedar
#

huh that's fun

#

...could that just be a network error potentially?

#

i'm not familiar with bass errors

still garnet
#

yeah i blame the internet

spark cedar
#

if you re-run does it happen again? could still potentially be release related, but it is super unlikely that we could have broken the last release as well

#

i'd actually be proud if we managed that 🙂

tepid nova
#

re-running

tepid nova
#

@spark cedar it went away

spark cedar
#

phew 😅

cosmic cove
#

docs PR fail

wild zephyr
tepid nova
#

(replied in the issue)

tender lava
#

Is the context directories available in 0.12.5?

tender lava
#

Additional question: Is Daggerverse already compatible with 0.12.5? I'm getting errors trying to publish (CLI) a module upgraded to that version to the Daggervese.

could not load module
I wasn't able to load a module from github.com/Excoriate/daggerverse/module-template@0dbf688a20cf568b9d1edd7acbd776b48ef0eeb6. Check the logs below for details.

get name: input: moduleSource.asModule resolve: failed to create module: select: module requires incompatible engine version: version v0.12.5 is greater than supported version v0.12.4
Creating new Engine session... OK!
Establishing connection to Engine... Setup tracing at https://dagger.cloud/traces/setup. To hide: export NOTHANKS=1

OK!
1   : connect
2   :   connecting to engine
2   :   | engine name=daggerverse-64d5cb5df5-7khrr version=v0.12.4 client=jn0vfcksdicjja461ucvcllp9
2   :   | cloud url=https://dagger.cloud/traces/setup
2   :   connecting to engine DONE [0.0s]
1   : connect DONE [0.0s]

3   : moduleSource(refString: "github.com/Excoriate/daggerverse/module-template@0dbf688a20cf568b9d1edd7acbd776b48ef0eeb6"): ModuleSource!
4   :   git://github.com/Excoriate/daggerverse#0dbf688a20cf568b9d1edd7acbd776b48ef0eeb6

Ref: https://github.com/Excoriate/daggerverse/actions/runs/10416963022/job/28850234715
https://github.com/Excoriate/daggerverse/blob/17e4a738a4ba33e68e0ae144cb0114e2f9452558/.github/workflows/cd-publish-last-version-dag-module.yml#L13

GitHub
🚀 Publish Latest Version of Specified Dagger Module · Excoriate/dag...

A collection of Dagger modules powered by Dagger. Contribute to Excoriate/daggerverse development by creating an account on GitHub.

GitHub
daggerverse/.github/workflows/cd-publish-last-version-dag-module.ym...

A collection of Dagger modules powered by Dagger. Contribute to Excoriate/daggerverse development by creating an account on GitHub.

spark cedar
spark cedar
#

Diving into some CI failures 🧵

tender lava
fair ermine
#

@civic yacht @spark cedar @still garnet Hey guys,

I did some research on the context directory issue and caught the problem, however, I don't know how to solve it because it's related to internal and dep module's execution environment.

I put a summary of my research there: https://github.com/dagger/dagger/pull/7744#issuecomment-2293510819

Could I get some help on this? Any idea is more than welcome.

/cc @tepid nova you can find more details about the issue I'm having there ^^^

GitHub
Feat/context directory by TomChv · Pull Request #7744 · dagger/dagg...

This is an implementation of #7647
API changes
This PR updates the Function and FunctionArg from our engine API to support defaultPathFromContext

Function

withArg(
""&...

tepid nova
fair ermine
tepid nova
fair ermine
#

Yeah, I need to find where to do that!

tepid nova
spark cedar
fair ermine
#

For context directory, it's pretty required :/

spark cedar
#

i'm confused though, if module A calls module B, then the context directory for B should be from A, not from whoever happens to be calling A
that would also play absolute havoc with our caching model - that means that it's unsafe to cache the call from A->B, right? because now it depends on a third external thing

fair ermine
#

So how can we solve the current context dir issue?

spark cedar
#

i think we can do it from the direct caller right?

fair ermine
#

The context of the direct caller isn't the host afaik, this is why I can see /scratch

spark cedar
#

right, i actually think that's probably the correct behavior 😛

#

it's uploading from the data inside the module right?

fair ermine
#

Yeah

#

But it's empty in that case, since there's no data inside the module itself

tepid nova
#

definitely the wrong behavior

spark cedar
fair ermine
#

Hmm

#

Limit context dir to module directly called from the host and not dependency, but I guess it's not an option

tepid nova
#

no not the host directory, the top level module's context directory

spark cedar
#

hm

#

yeah okay

tepid nova
#

or MAYBE the direct caller's context directory (debatable)

spark cedar
#

i'm not entirely sure what the way to implement this is gonna be

#

i'm also a bit worried, i feel like i'm really struggling to understand the implications of this, which to me is worrying because we have to document this/explain it to users.
i think i need to try poking around at this properly

#

maybe i need monday morning insight as well tbf, instead of tired friday thoughts

fair ermine
#

We can sync on this on Monday, no problem!

#

I have couple of other things to do

#

Sorry @tepid nova that this feature is taking such a long time to be merged :/

tepid nova
#

I feel like everyone is finally actually reading my issue

spark cedar
#

hm, okay, it doesn't seem infeasible to propagate the parent module source into module function's Call, but it's fiddly enough that it does feel weird

tepid nova
#

maybe re-read it one more time to be sure 😛

#

btw views are implemented wrong also

#

but that's ok because we'll deprecate them soon

#

don't worry about deadlines though @fair ermine . I only care about us getting it right

tepid nova
still garnet
#

I'll echo @spark cedar 's surprise that that's the behavior we want, only because it's inconsistent with all the other "host access" APIs whereby "host" always means the nearest local "host" i.e. the calling module.

Not saying it's right or wrong - the fact that Tom ran into it on his own seems to imply that's what we want - just pointing out that it will probably be a point of confusion if we don't document it properly. Also having trouble picturing a concrete example that hinges upon that behavior, since in my mind context dir was always sugar applied to the outermost layer, not something you'd really think about for dependencies. But I've been distant from the problem area for a while

wild zephyr
meager summit
#

hey folks, it seems like the integration tests are still hanging on CI/local setup. do we have a possible fix for it that I can try out?

spark cedar
meager summit
#

not yet, sorry 😦 looking into this now

little stone
#

Im running into a strange issue locally, just trying to run dagger functions

and running into this problem around the php sdk.

dagger functions
Full trace at https://dagger.cloud/levs-test-org/traces/2a5b42d1f0ee570cedac9d17e0384080

✔ connect 1.5s
✘ initialize 3.2s
! input: moduleSource.withContextDirectory.asModule resolve: failed to create module: select: failed to update module dependencies: failed to initialize dependency modules: failed to initialize dependency module: select: failed to initialize module: failed to call module "php-sdk-dev" to get functions: call constructor: process "/src/sdk/php/dev/entrypoint.php" did not complete successfully: exit code: 127
  ✔ resolving module ref 0.2s
  ✘ installing module 3.0s
  ! input: moduleSource.withContextDirectory.asModule resolve: failed to create module: select: failed to update module dependencies: failed to initialize dependency modules: failed to initialize dependency module: select: failed to initialize module: failed to call module "php-sdk-dev" to get functions: call constructor: process "/src/sdk/php/dev/entrypoint.php" did not complete successfully: exit code: 127
    ✔ ModuleSource.resolveFromCaller: ModuleSource! 1.7s
    ✘ ModuleSource.asModule: Module! 1.3s
    ! failed to create module: select: failed to update module dependencies: failed to initialize dependency modules: failed to initialize dependency module: select: failed to initialize module: failed to call module "php-sdk-dev" to get functions: call constructor: process "/src/sdk/php/dev/entrypoint.php" did not complete successfully: exit code: 127
Error: input: moduleSource.withContextDirectory.asModule resolve: failed to create module: select: failed to update module dependencies: failed to initialize dependency modules: failed to initialize dependency module: select: failed to initialize module: failed to call module "php-sdk-dev" to get functions: call constructor: process "/src/sdk/php/dev/entrypoint.php" did not complete successfully: exit code: 127

Stderr:
': No such file or directory

This is on a fresh clone of dagger/dagger on 0.12.5

meager summit
#

is there a way to reset the engine schema? I am trying to evaluate if I can install a module, run some test and reset the engine to run the second test

spark cedar
#

Im running into a strange issue locally

spark cedar
#

private modules via ssh auth is in! (huge kudos to @obsidian rover)

#

maybe worth cutting a release this week to bring it in? looks like context modules is also on the home stretch

obsidian rover
spark cedar
#

🧵 release discussion 🎉

tepid nova
#

nice @obsidian rover !!

spark cedar
#

👋 quick ask for when anyone has a spare moment - i've got a few easy-to-review eclectic open prs that need some eyes 🙂

wild zephyr
tepid nova
#

Guys I am getting bricked modules with 0.12.5

#

Repro:

$ dagger version
dagger v0.12.5 (registry.dagger.io/engine:v0.12.5) darwin/arm64

$ dagger -q -m github.com/shykes/sha functions
[...]
Stdout:
marshal: json: error calling MarshalJSON for type *dagger.GeneratedCode: input: container.from.withEnvVariable.withDirectory.withWorkdir.withMountedCache.withEnvVariable.withMountedCache.withEnvVariable.withExec.file resolve: process "go build -o /bass ./entrypoint" did not complete successfully: exit code: 1

Stderr:
# dagger/bass/entrypoint
entrypoint/main.go:38:8: not enough arguments in call to telemetry.Close
    have ()
    want (context.Context)
#

How do I downgrade to 0.12.4 on mac? I am stuck

tidal spire
tepid nova
#

I'll try that next

#

Just tried this (from our docs install instructions):

cd /usr/local
curl -L https://dl.dagger.io/dagger/install.sh | DAGGER_VERSION=0.12.4 sudo sh

And it actually always installs the latest

#

so the DAGGER_VERSION variable seems to be useless

little stone
tepid nova
#

@tidal spire OK getting the binary from github worked (MacOS complained a lot though)

tepid nova
tidal spire
tepid nova
tepid nova
twin crow
#

Two workflow are timing out constantly since yesterday: Engine & CLI / test and Engine & CLI / testdev (pretty sure they passed last week). Asking for getting my PR merged. Any pointer? It sucks that we end up ignoring those.

spark cedar
#

(sorry on mobile)

#

But you can pass -e to sudo to get DAGGER_VERSION in correctly (that flag forwards env vars correctly)

spark cedar
#

I guess we should have had module compat for this? But we've not had this case before, I hadn't really thought of that telemetry API as "stable" in that way before so missed this

#

cc @still garnet any thoughts on that? do we maybe need to start versioning our telemetry API?

still garnet
# spark cedar cc <@108011715077091328> any thoughts on that? do we maybe need to start version...

This specifically affects the Bass SDK, because a bit of its non-generated code depended on the telemetry API from the generated code, making it a bit of a moving target. I fixed it in the Bass SDK so that it depends on a stable library instead: https://github.com/vito/daggerverse/commit/d168f0a3422c4979f8ad5fe9f37ba3bc7f0df0a6 - but any modules using it will have to bump the (apko*) dependency for v0.12.5+. I'm not sure if module compatibility layer would be able to fix this or if it's "just" an artifact of the weirdness I mentioned around the Bass SDK having to contend with two different versions of Dagger

#

The dependency to bump is /apko - but, now that I think about it more, yeah the module compat. layer is probably the right solution regardless (in the interest of avoiding all possible breakage) - assuming it's possible

civic yacht
#

Two workflow are timing out constantly

tepid nova
#

@still garnet is there something special about apko that makes it break on 0.12.5 and not other dependencies?

#

(ie should I watch my other modules for something similar?)

still garnet
tepid nova
#

Ah so it's something about custom SDKs? Would all custom SDKs automatically break unless they update their telemetry code?

#

(or something like that)

still garnet
#

Normally the changed code interface is only used by generated code, so it's always in sync

#

Specifically the bass sdk had code that broke that assumption

tepid nova
#

Got it

still garnet
#

So literally just the bass sdk, which has ended up in a few dependency trees 😛

tepid nova
#

OK I updated the whole chain (DX note: pretty cumbersome to do that, maybe we can add features to make it easier). Should be good to go.

#

Tomorrow I'll have to track down all dependencies of my wolfi module, and get them to update.

#

So yeah it's definitely the kind of situation we were trying to avoid with compat mode I'd say

still garnet
#

yeah maybe we can just fix it in the next patch with compat mode

#

Retroactively

tepid nova
#

That would be sweet

still garnet
#

Would be good to keep working that muscle

tepid nova
#

In any case, I also got the feeling that this kind of laborious "mass update" would deserve some sort of hosted product to help me automate it. I would love for daggerverse to just hook up into all my upstreams and downstreams, and just keep things up to date for me

#

Also - run my module's tests (which don't exist at the moment)

#

Basically I need CI for my CI modules 🙂

meager summit
#

FWIW, I am adding a module to compare schema with different versions of dagger engine, so maybe we can add that check in CI of CI modules

obsidian rover
spark cedar
#

Compat for internal go code

tepid nova
tepid nova
wild zephyr
#

SDK initialization performance 🧵

#

SDK initialization performance 🧵

fresh harbor
#

I found an issue on the cli output if I do return a list of containers here https://github.com/dagger/dagger/issues/8202

Not sure what result should be, but I remember that shellcheck return list of struct so this should returns list of objects? Or I may do something wrong.

GitHub
🐞 cli: raise error when a function return a list of containers · Is...

What is the issue? From this snippet: $ cat main.go package main import ( "dagger/reproduce-bug/internal/dagger" ) type ReproduceBug struct{} func (m *ReproduceBug) Containers() []*dagger...

tepid nova
#

@civic yacht (low priority, just want to pick your brain on something you mentioned before).

You mentioned you had ideas to allow a module to test itself, without even requiring a special annotation like +dev or equivalent. I'm all ears!

Context: trying to wrangle the knot of DX issues around daggerizing an existing repo. Since we're discarding the "dev module" idea, it's becoming more important to make the DX really good with just one module. In particular:

  • Where do I put functions to test my module
  • Where do I put functions to generate the CI config that will wrap my module
civic yacht
#

@still garnet something extremely odd is happening w/ output when running tests locally:

  1. dagger call --source=.:default test specific --pkg="./core/integration" --run="TestModule"
  2. Let it run for a few minutes
  3. Ctrl-c to cancel
  4. It spends a minute or so printing insane amounts of output (can't tell for sure but might be a lot of duplicate output?), can't interrupt it at all
still garnet
#

@vito something extremely odd is

spark cedar
#

giving up for the day on this misery, upped the verbosity of the error even more https://github.com/dagger/dagger/pull/8214, and tried clarifying a few assumptions as well 😦 it feels so similar to the python error here where we can't find some file contents

tepid nova
#

@wet mason bug report in Directory.terminal:

dagger core directory with-new-file --contents=hello --path=/hello terminal
exit 1
civic yacht
# spark cedar giving up for the day on this misery, upped the verbosity of the error even more...

Yeah it feels very similar to the python bug and also to the ones in TestLegacy that were fixed... I'm starting to wonder if my fix there (which was to use the stable client ID) handled some subset of cases but left others around.

I am now setup with eks cluster access so plan is to just repro these errors in CI (since I can't locally) and then debug on those machines. At this point, it's just a matter of time+println debugs+strace and we should have answers 🙂

spark cedar
#

I'm also curious if you see that overlayfs error in kernel logs when it happens 🤔

#

"undefined behavior" in the filesystem could explain a whole host of issues

civic yacht
wet mason
#

Also maybe @twin crow can help — he just tweaked that code this week

tender lava
#

Hello 👋 , if I want to contribute to one of the core APIs (E.g., git. go in the /core package) like here https://github.com/dagger/dagger/blob/ea6489ef18302593539239174d7198829ef5ff90/core/git.go — where is the best place to boot a complete local environment, test the changes, etc.? The .dagger module seems to be a good place to start reading, but I found its documentation (https://github.com/dagger/dagger/blob/ea6489ef18302593539239174d7198829ef5ff90/.dagger/README.md#L1) a bit high-level for what I'm asking 🤔

GitHub
dagger/core/git.go at ea6489ef18302593539239174d7198829ef5ff90 · da...

An engine to run your pipelines in containers. Contribute to dagger/dagger development by creating an account on GitHub.

GitHub
dagger/.dagger/README.md at ea6489ef18302593539239174d7198829ef5ff9...

An engine to run your pipelines in containers. Contribute to dagger/dagger development by creating an account on GitHub.

meager summit
spark cedar
#

if you want a local env, you can also call ./hack/dev to build and launch the engine

#

then prefix commands to run in this dev environment with ./hack/with-dev

#

we should probably link to the ./hack/README from .dagger/README or CONTRIBUTING.md, etc

tepid nova
#

Questions on our CI:

  1. What is the 'dev engine' option and when do we use it
  2. What is the concurrency / group name tweak for
  3. What's the difference between the dagger-g2-v0-12-5-4c and dagger-g2-v0-12-5-16c-st-od runners
#

Also, why do we have GHA jobs that sequentially call several dagger functions, instead of just collapsing the sequence in a single dagger function?

#

What is the concurrency / group name tweak for

Answering myself: it allows "workflow preemption" so only one instance of each workflow runs at a time on each PR. Newer preempts the older. Only enabled on some workflows, presumably the most expensive?

fair ermine
spark cedar
#

Off on bank holiday today, but will take a look tomorrow 🎉

tepid nova
#

A fun sidenote. As I reverse engineer our own CI, here is my process:

  1. Find weird manual YAML. example: concurrency: and group:.
  2. Ask about it, read the code, figure out the intent (mostly figuring it out on my own, as people are busy and haven't had time to respond to me most of the time)
  3. Implement the intent as a new feature in my generator 😛 *example: https://github.com/shykes/gha/commit/a33ebda923541d21422069963d8cf287f0ed221b
  4. Rinse repeat

cc @tidal spire 🙂

GitHub
Configurable pipeline concurrency · shykes/gha@a33ebda

Signed-off-by: Solomon Hykes

fair ermine
#

@wet mason I did some work on the TS SDK, reordering the steps and it looks very promising!

https://github.com/dagger/dagger/pull/8236

@hasty basin If you have some time to play with that version, let me know if the TS SDK feels faster!
Normally, dependencies should only be downloaded if necessary but cached otherwise, I mount the source code after doing all the init/install job to trigger more cache hit, hope it helps!

fair ermine
fair ermine
#

@tepid nova I have a small question related to context dir with dep

If we have:

parent/
  dagger.json
  main.go
  dep/
    dagger.json
    main.go
.git

From the dep perspective, the / will root to the context dir, but the relative . should root to where? The parent module or ./dep?

#

I guess since we use the caller context, . should root the the parent module, that feel a big strange though

spark cedar
#

if you want the top-level, you should be able to do /

#

otherwise, you get very different results if you call the dep directly or through another module

fair ermine
#

Okay, I need to fix this then!

spark cedar
#

I think this is what @civic yacht means by "There's some fixes needed to ensure that the actual paths we use in each case are always based on the "callees" paths when a dep"

spark cedar
#

kudos to @meager summit for his work on the new modules/compatcheck module - now we can more confidently run a dagger module to work out when we've accidentally broken the daggerverse 🎉

tepid nova
#

but as usual we can change it if it's wrong

spark cedar
#

hm, just had a thought about TestPythonProjectLayout and why it might still be flaking 🤔

spark cedar
#

hm, ideally we could pass a unique hash of the . package? but this doesn't seem to have an effect from what i can tell.
we need different root paths for a unique module. maybe we could use the new .Digest functionality?

spark cedar
wet mason
wet mason
#

Go / Python / Typescript [/cc @rancid turret]

Looks like they all have the same problem of loosing all cache as soon as anything chnages in the source

Screenshot_2024-08-27_at_7.12.51_PM.png Screenshot_2024-08-27_at_7.13.03_PM.png Screenshot_2024-08-27_at_7.13.11_PM.png
fair ermine
#

Or maybe I need to combine both parent context to get the client metadata but also use the current context to get data about the children ctx, that might be the way

civic yacht
#

. should just refer to the module source dir right? We know that path relative to the context root, it's stored as part of ModuleSource. So ModuleSource.LoadContext should be able to resolve any . paths based on that source dir path

fair ermine
#

I'm checking if I can use sourceRootPath to append it

civic yacht
#

So if the wrong dir is being loaded then incorrect values for paths are being supplied somewhere

civic yacht
#

I poked around a bit on your PR locally @fair ermine, it seems like the issue may come down to the fact that src.SourceRootRelSubPath() is incorrectly returning . on the dep module: https://github.com/TomChv/dagger/blob/0e3ab2e05c6ef4e28ec6f79b6188bcc12d96ff88/core/modulesource.go#L400

Will let you go from there, but that is what's leading to the wrong module source dir being used.

GitHub
dagger/core/modulesource.go at 0e3ab2e05c6ef4e28ec6f79b6188bcc12d96...

A portable devkit for CI/CD pipelines. Contribute to TomChv/dagger development by creating an account on GitHub.

fair ermine
tepid nova
civic yacht
fair ermine
#

True!

tepid nova
#

go sdk: test
The self-hosted runner: dagger-g2-v0-12-5-4c-6822z-runner-gbh4m lost communication with the server. Verify the machine is running and has a healthy network connection. Anything in your workflow that terminates the runner process, starves it for CPU/Memory, or blocks its network access can cause this error.

fair ermine
bronze hollow
#

YAY

#

Context Directory 🎉

meager summit
#

Hi @spark cedar I have a question regarding how views are used on client side and engine side 🧵

tepid nova
#

Feature request: better middleware support.

It's not easy to wrap an existing module with many optional arguments, and simply inject / change a few arguments while passing through everything. In go I want to take the ... dagger.OtherModuleFuncOpts and pass it through, but I can't. I have to copy-paste every single argument, and manually plumb them through one by one

#

Fun fact I implemented our self-hosted runner naming logic in a dagger function 🙂 Thank you @astral zealot for explaining it all to me.

Example:

dagger call -m github.com/dagger/dagger/.github@pull/8241/head \
 runner --cpus=16 --dagger-version=v0.12.5 --dind=false --generation=2 --no-spot --single-tenant

I also reverse-engineered our existing runner configurations, and took the liberty of packaging them 🙂

INTRODUCING.... The bronze, silver, gold and platinum runners!

$ dagger functions -m github.com/dagger/dagger/.github@pull/8241/head
[...]
bronze-runner     Bronze runner: Multi-tenant spot instance, 4 cpu
silver-runner     Silver runner: Multi-tenant spot instance, 8 cpu
gold-runner       Gold runner: Single-tenant on-demand instance, 16 cpu
platinum-runner   Platinum runner: Single-tenant on-demand instance, 32 cpu

Example: "give me a platinum runner with docker-in-docker, top of the line"

$ dagger call -m github.com/dagger/dagger/.github@pull/8241/head platinum-runner --dind
${{ github.repository == 'dagger/dagger' && 'dagger-g2-v0-12-5-32c-dind-st-od' || 'ubuntu-latest' }}
astral zealot
#

Nice!! We used to have 64c runners, would those be Adamantium? Or vibranium for the marvel fans 😆

tepid nova
spark cedar
#

Is it ready to review yet? 😍
and any thoughts on moving the gha module into dagger/dagger? or do we want to keep it separated out?

spark cedar
#

ahhh it's so cool

#

i'm just ever-so-slightly annoyed i didn't get to work on this 😛 it's cool

#

left some high level ideas on just glancing through, but i think i'm biased towards merging soon (next week? 👀) - i can carve out some time to help as well if needed
i really wanna give this approach a real go

rancid turret
tepid nova
tepid nova
tepid nova
#

I had to try

#

Everything is queued 😦

astral zealot
#

Lets see the world burn daggerfire elmofire

tepid nova
civic yacht
civic yacht
#

omfg I finally caught one of the mystery flakes in action and manually confirmed something terrible is happening involve the go build cache:

fail to build due to an import not being found:

buildkitsandbox:/src# /usr/local/go/bin/go build -o /runtime .
internal/dagger/dagger.gen.go:23:2: package dagger/bare/internal/querybuilder is not in std (/usr/local/go/src/dagger/bare/internal/querybuilder)
internal/dagger/dagger.gen.go:24:2: package dagger/bare/internal/telemetry is not in std (/usr/local/go/src/dagger/bare/internal/telemetry)

But when looking at the file it's erroring on, the lines it's complaining about don't match the error at all:

buildkitsandbox:/src# head -n 100 internal/dagger/dagger.gen.go 
// Code generated by dagger. DO NOT EDIT.

package dagger

import (
        "context"
        "encoding/json"
        "errors"
        "fmt"
        "net"
        "net/http"
        "os"
        "reflect"
        "strconv"
        "strings"

        "github.com/Khan/genqlient/graphql"
        "github.com/vektah/gqlparser/v2/gqlerror"
        "go.opentelemetry.io/otel"
        "go.opentelemetry.io/otel/propagation"
        "go.opentelemetry.io/otel/trace"

        "dagger/test/internal/querybuilder"
        "dagger/test/internal/telemetry"
)

Then, rm -rf the whole go build cache and now it magically builds again!!

buildkitsandbox:/src# rm -rf /root/.cache/go-build/*
buildkitsandbox:/src# /usr/local/go/bin/go build -o /runtime .

I don't see any reasonable explanation other than some bug in go that corrupts the build cache

civic yacht
hasty basin
#

@spark cedar if I have a v0.12 module with a v0.10 or v0.11 dependency, how should I think about how that executes?

spark cedar
#

🤔 how do you mean sorry

#

it should execute "as expected" - each module gets presented a schema that is compatible with it's declared engineVersion

hasty basin
spark cedar
#

yeah, so if your deps have deps, then they all still get served the right versions as well

hasty basin
spark cedar
hasty basin
#

Source of my question: I had sent a PR to @astral zealot for his Pulumi module (merged and working fine) and at the time had glanced at the dagger.json deps and thought the sha of the docker modu'e dep "looked like" the sha for the latest version of Solomon's docker module, but it was an old v0.10.x version. I need to send another PR eventually, but it made me think, hmmm....if the dep's API changes, of course you have to accomodate, and you need to consider any Dagger API changes.

I think maybe Rajat's work can help here, eh @wild zephyr ?

spark cedar
#

but for example, a module might return a container - even if that container from the perspective of your module has method, if that method was removed in the version of the caller, then it's only the caller who doesn't see it

#

the callee still gets to see it and it's fine

hasty basin
spark cedar
#

exactly

#

honestly, i'm just really impressed that the design of the module compat stuff has held up, there was always the potential that some insane case would appear that just was fundamentally broken (and even the strcase work has kinda been doable with it)

spark cedar
#

🧹 hopefully one flake down

rancid turret
tepid nova
meager summit
hasty basin
#

deps compat checks

tepid nova
#

I'm observing some differences between our +ignore behavior, and the .gitignore behavior

#

Which we should fix quickly (if they are real) before people start relying on the current behavior

civic yacht
# civic yacht Put the above GH issues here: https://github.com/dagger/dagger/issues/8031#issue...

Managed to get this reproducible in a docker image by tar'ing up one of our containers in CI that hit it, used that to open an issue in the go repo: https://github.com/golang/go/issues/69179

Commands to repro are in there if anyone else wants to poke around at it!

GitHub
cmd/go: go build errors that don't correspond to source file until ...

Go version go version go1.23.0 linux/amd64 Output of go env in your module/workspace: GO111MODULE='' GOARCH='amd64' GOBIN='' GOCACHE='/root/.cache/go-build' GOENV=&#...

tepid nova
#

What does dagger call dev-export do?

#

It looks new

#

"creates a static dev build" so it's dagger call build?

spark cedar
#

sort of, it builds the cli and the engine together - bundles them up and throws them in bin together. the reason is, building them separately was resulting in loading the entire dev module twice (which is a bit slow, perf investigations are ongoing with cache, etc)
ideally, we'd solve this with a "dagger script" - but we still don't really have that yet

#

⚠️ think we need to release v0.12.7 today - pretty major regression found in https://github.com/dagger/dagger/issues/8291
essentially, doing dagger install github.com/foo/bar@version no longer works - working on a fix now
EDIT: v0.12.7 has been released and resolves the issue

GitHub
🖐️ dagger install fails · Issue #8291 · dagger/dagger

What happened? What did you expect to happen? I'm probably doing something wrong but i cant seem to figure out what is going wrong. When i try to install the example from dagger install --help ...

spark cedar
#

@wild zephyr do you have push access to github.com/shykes/hello? i know we want to avoid pointing people to it, but can we get a tag for v0.3.0 (or v0.2.1) to point to the current head? the last tag v0.2.0 cannot build in a recent version of dagger (this was before mod-compat)

wild zephyr
#

No, I don't have access 😦

spark cedar
#

ah, no worries

#

i wonder if we should have a github.com/dagger/hello or something that we can call during our quickstart? then we have an "official" repo to point to, since users might not be aware of who we all are, and dagger definitely feels much more official

rancid turret
#

Working now: dagger functions -m github.com/shykes/hello@v0.3.0

spark cedar
tepid nova
#

@spark cedar @wild zephyr I can give you access

#

Done (and also yes to dagger/hello)

spark cedar
#

Thank you!

tepid nova
#

@spark cedar do you think we can merge 8241 today? 🙏

#

the conflicts are nothing too big (0.12.7 bump today) but it does take a little time to deal with

#

@fresh harbor thanks for looking at the elixir error! Do you think the YAML config itself could be responsible for the cache volume error?

tepid nova
#

UX nit: can we please rename dagger core dagger-engine to dagger core engine? 🙏 Also keep me in the loop for API review so I can weigh in on aesthetics 😛

#

same thing for blob... Include me in design review people please

#

Fun fact, if I call dagger core dagger-engine local-cache, it includes prune in the auto-printable fields list... Which means it actually prunes your cache

tepid nova
tepid nova
#

failed to get configured module: failed to get module ref kind: Post "http://:mem/query": context canceled

#

Error: response from query: Post "http://:mem/query": command [docker exec -i dagger-engine-d522a778c2389390 buildctl dial-stdio] has exited with exit status 137, make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=

#

Smells like an OOM? But I have plenty of ram on the docker VM. And never hit this issue before

Screenshot_2024-09-03_at_5.44.17_PM.png
fresh harbor
spark cedar
#

same thing for blob... Include me in

#

v0.13.0 release thread 🧵

fresh harbor
#

I found an error during call dagger develop on main:

$ dagger develop

Stdout:
marshal: json: error calling MarshalJSON for type *dagger.GeneratedCode: input: container.from.withEnvVariable.withDirectory.withWorkdir.withMountedCache.withEnvVariable.withMountedCache.withEnvVariable.withExec.file resolve: process "go build -o /bass ./entrypoint" did not complete successfully: exit code: 1

Stderr:
verifying github.com/klauspost/compress@v1.16.5/go.mod: checksum mismatch
        downloaded: h1:G7mAYYxgmS0lVkHyy2hEOLQCFB0DlQFTMLWggykrydY=
        go.sum:     h1:ntbaceVETuRiXiv4DpjP66DpAtAGkEQskQzEyD//IeE=

SECURITY ERROR
This download does NOT match an earlier download recorded in go.sum.
The bits may have been replaced on the origin server, or an attacker may
have intercepted the download attempt.

For more information, see 'go help module-auth'.
spark cedar
#

hm

#

i do not see this 🤔

#

potentially could be some network corruption? does it happen on retries?

fresh harbor
#

Oh, let me try disable corporate proxy first.

spark cedar
#

is your coprorate proxy "an attacker" 😄

fresh harbor
#

Work after turn off. So scary. 😱

meager summit
rancid turret
tepid nova
#

Anyone still around to help me get 8241 and 8284 through the finish line?

tepid nova
#

I have a bunch of failing checks now, no idea why.. flake, no flake?

obsidian rover
tepid nova
#

Actually I was able to reproduce that one

# 
dagger call -m github.com/dagger/dagger@pull/8284/head check --targets=sdk/go
#

There's a failing git clone but I have no idea where it comes from

#

fatal: repository '' does not exist

#

ooh I think I know what it is

#
  1. This is very hard to debug because of lack of self-calls
#
  1. I just recently added TestPublish
#
  1. I think I forgot to pass an important argument to TestPublish it needs a specific tag I think
#

This would actually be immediately made better by splitting out each sdk-specific pipelines into sub-modules

obsidian rover
tepid nova
#

btw if we're looking for performance optimizations... our pipelines are still litered with ad-hoc from("alpine").withExec(["apk", "add", "random", "crap"])

obsidian rover
#

how come would it be ""

tepid nova
#

@obsidian rover it's also because of lack of self-calls. I added a top-level function (Check) that wraps the SDK's Publish functions (among other things). Because it's an intra-module call, I lose all the default values of optional arguments

#

So I have to either move everything into a sub-module (useful but lots of work), or copy-paste all the values from the +default pragma

obsidian rover
tepid nova
#

I'm unblocked, will try to power through the gruntwork before my cutoff, thanks!

#

OK, breaking out the SDK sub-modules is even worse than I thought, because the SDK pipelines depend on Engine().installer(), so to avoid dependency loops I also need to break out an engine/ submodule

#

copy-pasted default arguments it is

#

I'm having a lot of trouble with +ignore

#

it seems pretty badly broken

Update: it's specifically includes (!foo) that seem to break

tepid nova
wet mason
spark cedar
#

i don't really know about that, i feel like if the image isn't local, and we can't pull it, that should be a hard failure 🤔

wet mason
#

Yeah ... I was wondering if there's a weird dev case where it's used

spark cedar
wet mason
#

However, I don't know if there are side effects associated with that, especially in "non production" (e.g. ./hack/dev etc)

wet mason
spark cedar
#

we don't hit this path at all in ./hack/dev actually 😆 we're hitting the other "engineconn" driver, since we've got docker-container:// in the url

#

this is only for docker-image://

wet mason
#

huh

#

wondering what all scenarios are ...

#

currently that code handles images with digests directly, otherwise looking up the digest, fallback etc

spark cedar
spark cedar
#

this is like asking for "v0.13.0", and fails to pull, so you fall back to "v0.12.7"

wet mason
# spark cedar imo, i think 8340 is cleaner as an approach, i like that it keeps the image id i...

I thought so too ... but deep down, I think it's meaningless?

In the sense that it's not really content addressed at the end in any case

Since we're conditionally pulling and resolving the digest at different stages, there's no guarantee that we're running the same thing

e.g.

  1. We docker pulled v0.12.7 with digest X
  2. THEN, we get the digest of v0.12.7 as digest Y
  3. We'll create a new container Y, not run a docker pull, and just end up with running digest X named Y
#

false sense of content addressability

#

at the end of the day, what matters in that logic is the tag name regardless.

#8341 makes it so containers are named accordingly rather than thinking they're more content addressable than they really are

spark cedar
#

mm, this does improve container naming as well

wet mason
#

yeah, what I was thinking is: in the end, they're the same (one does not offer more guarantees than the other). But the latter happens to be less code and shaves off ~half a second on first run (when we don't have an image locally and need to talk to the registry)

spark cedar
#

right i get it now

#

sure, i agree now, i prefer approach 2 😄

wet mason
#

"first run" happens to be either when someone first looks at dagger, or in CI (e.g. GHA) for each build when it's ephemeral runners, so that half second is pretty important

spark cedar
#

@rancid turret @fair ermine I have some questions on private properties 🙂

tepid nova
#

so the performance benefit is even greater than that

#

this error again... @spark cedar

spark cedar
#

i don't currently have any capacity to dig properly into it sorry

tepid nova
#

No worries, just wanted to add to your list

bronze hollow
#

Search is failing me, is there an open issue for syncing a directory for local development?

tidal spire
spark cedar
#

Your wish is my command 🙏

tepid nova
spark cedar
#

not sure about engine lint 🤔 looks like the job got cancelled? at the infra level?

#

one of the commits needs a sign-off, which is why DCO is failing

tepid nova
#

pushed with java test-publish omitted

#

I don't see an missing signoff?

spark cedar
tepid nova
#

I've been through so many loops with flakes, after a while you start assuming everything is a flake

#

(yes it is me, captain obvious)

tepid nova
#

Looks like the checks are passing! Although this is awfully slow... the github UI hadn't refreshed, checks were already done

#

All checks are green, all reviews have been addressed... Last call before merge 🙂

tepid nova
#

merged! Our pipelines officially use context directory 🙂

hybrid widget
tepid nova
#

You can ignore the one on CI integrations for now, it's pretty decoupled

#

Also @little stone started sub-issues to allocate those invididual review tasks, we're going to divide and conquer with @tidal spire @hasty basin @wild zephyr @little stone

tepid nova
#

Problem: when loading a module from git, the .git directory is not available in the context... Any way we could change that?

#

@still garnet telemetry explosion of some kind?

still garnet
#

looks like a network flake? the giant error barf is because we currently yeet stdout/stderr into error messages, which is part of the in-flight work I'm trying to land

tepid nova
#

I'm re-running, let's find out!

#

Context dir in action: open an interactive terminal with a dev build of any PR ready to go:

dagger call -m github.com/dagger/dagger@pull/8353/merge dev terminal
tepid nova
tepid nova
#

God our CI codebase is a tarpit

#

the curse of early adopters

spark cedar
#

the job "succeeds", and you get this super helpful little email from github

image.png
#

"pr failed" but "all jobs succeeded"

spark cedar
#

i had a use case for wanting the alpine module somewhere else, so wanted to get it in asap

#

(tldr, trying to tidy up issue/pr backlog of things this afternoon before i head off on a couple days of pto next week)

tepid nova
tepid nova
spark cedar
#

it only appears to be java for now, but not quite sure, i'm unfamiliar with the step summary

tepid nova
#

oh! I did see that yesterday

#

it's the markdown file I generate. it contains all stderr , so that hit a file size limit I wasn't aware of

meager summit
#

Hello folks, I am making changes to make resolving Directory lazy (ref https://github.com/dagger/dagger/issues/6713), and running into a scenario where I am not sure where to actually check for the existence of dir:

            From("alpine:latest").
            Directory("/dont-exist").ID(ctx)

in this case the id is returned from the server (a base64 string).

what would you expect to happen here after the Directory is made to resolve in a lazy way

tepid nova
#

Hello folks, I am making changes to make

#

Would anyone be available to help me figure out interfaces? I can't get it to work

#

@tidal spire @wild zephyr any chance I could pick your brains for a few minutes? I'm really stuck

wild zephyr
#

let's go to #team-audio in the meantime

tepid nova
#

The conclusion so far is that interfaces are unusable in practice

#

Would love someone to contradict me

wild zephyr
#

@tepid nova found my example. It's on v0.10.x. Let me bump it to v0.12 to validate if it's still supposed to work the same way

tepid nova
#

Thanks - we made some progress with @tidal spire, jumping on a call but will resume after. I think unlocking interfaces would be very valuable for all of us

wild zephyr
#

have 5 min to show you?

tepid nova
#

No I'm in another call now.. sorry. will try to free up soon

wild zephyr
tepid nova
wild zephyr
tepid nova
#

will be back in a few mn

wild zephyr
#

I'm in the team audio chat

tepid nova
#

@wild zephyr @tidal spire I got so close...

# 
func New(
    // +optional
    // +defaultPath="/"
    // +ignore=["!sdk/go"]
    source *dagger.Directory,
    engine dagger.TypesSidecar, // --> the interface, defined in another module
)
# 
Error: get module name: input: failed to get schema: failed to get schema for module "go-sdk": failed to install constructor: failed to create function: failed to find mod type for function "" arg "engine" type
#

I am wondering if maybe, we don't support receiving as an argument an interface defined in another module

#

(yes I have 4 modules now)

#

Because unlike the app/platform example, I don't have N "apps" implementing the interface: I have N "platforms" receiving the interface as argument

#

Confirming my suspicion: if I change the code to use my own interface definition as argument (like in the example), it works:


// I don't want to define this here (duplication across SDKs)
// But it seems I have to
type Sidecar interface {
    dagger.DaggerObject
    Bind(ctr *dagger.Container) *dagger.Container
}

func New(
    // +optional
    // +defaultPath="/"
    // +ignore=["!sdk/go"]
    source *dagger.Directory,
    //engine dagger.TypesSidecar,
    engine Sidecar,
)
#

@obsidian rover @fair ermine @still garnet do you know by any chance if this (#maintainers message) is possible?

wild zephyr
tepid nova
#

@still garnet that error comes from a line you edited last, so I still have hope 😛

tepid nova
wild zephyr
#

Oh, I see.. Can't use use the interface directly then if you add the package as a Go dependency?

#

I'm just guessing here. Not exactly sure how this works under the hood

tepid nova
#

If the only way to reuse definitions is by a go import, that makes it less useful IMO

#

it means messing with go mod replace etc

#

could be messy too, because the imported interface will embed a different DaggerObject from its own dagger.gen.go

tepid nova
#

(confirming that dagger modules can't reuse each other's types via go imports, because they re-generate their own version of all core types)

wild zephyr
tepid nova
#

All roads lead to .dagger/internal/build

meager summit
fresh harbor
bronze hollow
#

Has there been any discussions on how we map dependencies when using functions?

When we expose everything as a function we can only map dependencies; we lose all visibility of dependants.

#

Are we getting one step closer to dagger.json being our BUILD file? 😁

tepid nova
#

Yeah I think ideally, your Dagger dependency graph models your project's dependency graph

#

@bronze hollow I'm actually working on implementing this pattern in our own repo right now

bronze hollow
#

I want to have my monorepo 100% Dagger by end of Sept so happy to discuss ideas to make the DX slick

tepid nova
#

Yeah same here 🙂 We finally have the features we need

#

I'm learning a ton on monorepo best practices as I do this, will braindump after

tepid nova
#

@bronze hollow here's a little teaser... That branch is a huge mess, the overall project doesn't build. But, I started spinning out parts of the monorepo into self-contained modules... And as a result, you can already load and consume those 🙂 If you're familiar with Bazel or Nix... this is starting to encroach in their model of packaging and distribution.

This will build our codegen tool straight from my PR:

dagger call -m github.com/dagger/dagger/cmd/codegen@pull/8355/head build --platform=current -o .
./bin/codegen --help
spark cedar
#

I'm a little worried about the perf overhead of splitting out so many modules? Hopefully we can get somewhere significant with the perf investigations soon, I've definitely felt the slowdown from the modules we've added right now (since we started several months ago)

Definitely shouldn't stop us, just curious if you've observed anything from a perf perspective so far?

tepid nova
#

Too early to tell

#

But in our case, I expect the enormous amount of complexity we can remove, will make it overall faster

#
  • de-duplicating base images and system package installs
#
  • vastly improved local dir filtering, which should bring upload times down
#

But yeah, module loading needs to be faster. If we feel some pressure on that point, probably a good thing

spark cedar
#

Mm yeah definitely, good point on the dir filtering especially

tepid nova
#

I'm already feeling it during local dev... It is quite pleasant!

spark cedar
#

I think I have some organizational questions as well, but probably best to hold those until we have something that's a bit more broken up
E.g. where does the go version get set? Is it set centrally in the go module? If so, that does mean it's less useful for consumers that aren't directly us, but if not, then we need another place to manage it

#

Or maybe we can infer it from the go.mod of the project root (which would be pretty cool)

#

But I guess the same question applies to things like alpine versions

tepid nova
#

Yeah I didn't get to that part yet. But I am also wondering

spark cedar
#

Oooh do you know what would actually be neat. If in my dagger.json, I could have dependency parameters, that were automatically passed to the constructor? Then I could say I want this version of the Alpine module, that installs "edge" or another version

#

You could be able to annotate your constructor params with a pragma to indicate that you could do that

tepid nova
spark cedar
#

Yes 🙂

#

Didn't know about that one 😔

tepid nova
#

It's old... and obsolete also. But the idea has been floating around

spark cedar
#

I think if you dagger install had args like dagger call? And you could pass string/int/etc args to it, that populated your constructor default

#

Then it's just part of the single source of truth, dagger.json

#

I think it's important that those parameters are somehow caller defined - if it's callee defined, then the go module we write for dagger can't be reused in another project as easily, and so the ecosystem becomes that little bit more fragmented

tepid nova
#

Here's what I think. On the topic of daggerizing big monorepos, we had been stuck for a long while. Right now, this feels like we're demolishing a wall, and finally making forward progress again.

I think in a couple weeks, once the dust settles from removing the obstacles immediately in front of us, we will mentally move to the next set of problems - and I expect we will make rapid progress fixing those too.

TLDR: we have momentum again on monorepos

spark cedar
spark cedar
tepid nova
spark cedar
bronze hollow
#

I added my constructor method and realised it broke codegen / module discovery

bronze hollow
bronze hollow
spark cedar
#

I don't think we need to totally rethink the approach just yet, there's a lot of benefits to distributing through git instead of oci

bronze hollow
bronze hollow
tepid nova
#

Observation: the next bottleneck for monorepo will be self-calls.

Before context dir, it was an inconvenience. Now it makes some patterns actually impossible, because when you call a function within the same module, you don't get context directory access

bronze hollow
#

When using dagger install, it would be nice to have a short-hand for a relative path from the Git root.

dagger install /dagger/typescript or dagger install git://dagger/typescript

bronze hollow
tepid nova
#

👋 @wild zephyr @rancid turret @still garnet with context dir not allowing access to the caller context, this "platform module" pattern will not be as nice. IMO we should either fix it, or stop recommending it. wdyt?

#general message

still garnet
#

platform modules

bronze hollow
#

I want to see if this improves my monorepo tasks 🙂

bronze hollow
tepid nova
#

then test in that terminal

#

I think there's a dev export too which you can save to the local filesystem, but not sure how to set it up after

wet mason
#

@stray heron @spark cedar [/cc @wild zephyr] For the CLI to support a "version check" (e.g. nag when a new version is available), what are your thoughts on:

  • Publishing a engine:latest tag
  • Including a version field in the manifest with the actual published version
  • ... so that the CLI can grab the manifest for latest, extract the version and compare against the local engine.Version const? (assuming it's not a dev engine, yadda yadda)
wild zephyr
rancid turret
#

Context directory ignore patterns

spark cedar
wild zephyr
spark cedar
#

Comparing against engine.Version, also good, there should already be a helper to detect if there's a dev version present

#

...back to holiday now 🏖️ 🍻

wild zephyr
tepid nova
stray heron
still garnet
#

If anyone has a moment, could use a quick review on this PR: https://github.com/dagger/dagger/pull/8384

I say "quick" because it just moves code around with no semantic change. Not sure why so many CI things failed, looking into it, but seems unrelated. This is a precursor to WASM + the other work I had in flight, to make those future PRs less confusing

still garnet
#

^ all green now

obsidian rover
tepid nova
tepid nova
tepid nova
#

In case it's useful to someone:

function dagger-dev() {
 set -x
 script=$(mktemp); echo "dagger $@" > $script
 dagger call -m github.com/dagger/dagger \
   dev \
   with-mounted-file --source=$script --path=/entrypoint.sh \
    with-mounted-directory --source=. --path="" \
    with-exec --args=sh,-x,/entrypoint.sh \
    stdout
}

Example:

$ dagger core version
v0.12.7
$ dagger-dev core version
v0.13.0-010101000000-dev-90e2ebfc2ec9
#

Does anyone know how to do a 0.12.8 release? Would be nice to get this out quickly, context dir is unusable without it on real monorepos (@bronze hollow and myself both affected)

tepid nova
#

I'm getting this error while building the CLI:

[...]
go: downloading github.com/morikuni/aec v1.0.0
go: downloading github.com/tonistiigi/go-csvvalue v0.0.0-20240710180619-ddb21b71c0b4
go: downloading github.com/tonistiigi/vt100 v0.0.0-20240514184818-90bafcd6abab
go: downloading github.com/tonistiigi/units v0.0.0-20180711220420-6950e57a87ea
go: downloading golang.org/x/time v0.3.0
go: downloading github.com/docker/distribution v2.8.3+incompatible
go: downloading github.com/moby/locker v1.0.1
go: downloading github.com/google/go-cmp v0.6.0
go: downloading github.com/containerd/stargz-snapshotter/estargz v0.15.1
go: downloading github.com/klauspost/compress v1.17.9
go: downloading github.com/containerd/ttrpc v1.2.5
go: downloading github.com/secure-systems-lab/go-securesystemslib v0.4.0
go: downloading github.com/shibumi/go-pathspec v1.3.0
go: downloading github.com/containerd/stargz-snapshotter v0.15.1
go: downloading github.com/vbatts/tar-split v0.11.5
error obtaining VCS status: exit status 128
    Use -buildvcs=false to disable VCS stamping.

Does this ring a bell for anyone?

obsidian rover
# tepid nova I'm getting this error while building the CLI: ``` [...] go: downloading github...

Using dagger or directly with go build ? I think you encountered it recently last week no ? It's when Go uses the "vcs ping" command to retrieve info on the repo. We do rely on it to resolve the ref, but the log would be wrapped with some other output, so I think it's the error coming from go itself here

I presume you're building outside of the git, or the "git remote ls" command fails in one of the context where you try to build. Is the .git present at the moment of failure ?

tepid nova
#

(note on Go SDK DX: to make this change, I have to manually edit 5 identical ignore pragmas)

tepid nova
#

Is there a way to get my current module's path relative to the context?

obsidian rover
#

as an API in an sdk ?

tepid nova
tepid nova
bronze hollow
wet mason
#

Just noticed with @stray heron that on init, the parent directory (which contains .git) is being uploaded -- is that by design or bug?

stray heron
meager summit
stray heron
wet mason
fair ermine
#

Is it because it actually uploads the whole repo to the engine? I got that issue on my playground repo that have 150K files in

rancid turret
#

RequiredPaths

tepid nova
#

@rancid turret if I wanted to look at the code that does the "function selection" for auto-printing an object, where should I look?

rancid turret
bronze hollow
#

Is there a published JSON Schema for dagger.json, or should I open an issue?

obsidian rover
bronze hollow
#

I see 0.13

wet mason
spark cedar
#

also noticed, because every job is separate, re-running failed jobs when things get cancelled like in #infrastructure message, means clicking into every single job and clicking the re-run button (instead of being able to just re-run all failed)

wet mason
#

👋

SDK re-generation triggers a CI failure for PHP (different indentation)

Am I doing something wrong or is something broken and just happened to land on me?

dagger --source . call sdk all generate export --path .

spark cedar
#

hm, i have seen this before (i'm not seeing it)

#

i remember fixing this by git cleaning the sdk/php dir

wet mason
#

I'll try, thank you!

#

@stray heron btw -- is there a way using test specific to run a single test of a suite? (e.g. not the entire suite)

rancid turret
#

Yep: --run=Go/Init

spark cedar
#

beat me to it 😄

#

you can also do regexp between each set of //

#

which is pretty handy

rancid turret
#

The Test prefixes aren't necessary.

wet mason
#

life changing

#

🙏

spark cedar
#

you can keep adding /s btw, if you have subtests in t.Run 😄

#

very useful for some of the tests that have about a 100 subtests 😆

wet mason
#

I was waiting 8+ minutes between runs of my little test

Screenshot_2024-09-11_at_4.51.11_PM.png
tepid nova
spark cedar
meager summit
bronze hollow
#

Nested Modules / Context Arguments

still garnet
#

@fair ermine (portalfrom #1283455682112393227 message) It seems like there's a bug with the TypeScript SDK's OTel stack:

dagger-dev call -m github.com/RawkodeAcademy/RawkodeAcademy@feat/dagger bun-install

When I run this, I don't see any activity beneath .bunInstall - just internal blob() calls. Here's the implementation: https://github.com/RawkodeAcademy/RawkodeAcademy/blob/3ab311052a09fcb3d65da63f7ae528811252c0a8/dagger/src/index.ts#L31

GitHub
RawkodeAcademy/dagger/src/index.ts at 3ab311052a09fcb3d65da63f7ae52...

Rawkode Academy Monorepository. Contribute to RawkodeAcademy/RawkodeAcademy development by creating an account on GitHub.

tepid nova
tepid nova
#

If I move install.sh and install.ps1 to a scripts/ subdirectory, will anything break?

#

or even installer/

#

It's not like that script is useful in the context of a source checkout - it feels more like a piece of the release infrastructure

hasty basin
#

So if they are moved and we update the RELEASING.md with their location, should be good.

tepid nova
#

Maybe we should have a release/ directory with all related files?

hasty basin
#

I'll defer to others closer. But there aren't many other files. Mostly other repos and such to deal with during a release, so installer/ makes sense to me.

tepid nova
hasty basin
meager summit
#

Hi Folks, when calling Up on a *dagger.service from within the module sdk, it does not seem to open tunnel to the host, but works if the module return *dagger.Service, and we call up from the CLI, is that expected?

spark cedar
#

it's just manually uploaded for now, but @stray heron will know about whether there's been any progress on automating that

wet mason
#

@fair ermine Hey btw, did we add a cache mount for corepack I don't remember?

bronze hollow
#

Does dagger install of a Git local module actually mount the entire Git repository and then reexport it?

obsidian rover
still garnet
spark cedar
#

hm

#

cc @fair ermine

wet mason
fair ermine
fair ermine
fair ermine
spark cedar
#

^ @fair ermine have a repro

#

oh i guess this is technically intended 😛

fair ermine
#

Yes actually haha

#

The ! doesn't act as "include only" but instead negate a potential more global ignore

#

That what Solomon actually asked me to support based on the issue he wrote

+ignore=["!.github/workflows"] -> nothing is ignored

still garnet
#

Yeah I think we just missed a spot in applying the fix

fair ermine
#

Btw I'm working on the "ignore metadata" for any dir argument, I got a quick question:

From a converted dagger value (from convertToSDKInput), can I cast the value to dagger.Instance[*Directory] or *Directory?
If so, is there a better way than creating a scratch dir to copy the dir with ignored pattern to ignores files?

Because I feel like dag.directory().WithDirectory(MYConvertedDir, {ignore: xxxx}) isn't pretty at all

#

Maybe I could do it from the CLI, @rancid turret where do we convert a Directory path to an actual dir? That might be a better way

#

But modSrc.ResolveDirectoryFromCaller has no ignore support, so I wonder what's the best idea

#

Oh I have an idea

spark cedar
fair ermine
spark cedar
#

it needs doing in both

#

it needs to be done in the cli as an optimization

fair ermine
#

No but I mean, when I send argument to ResolveDirectoryFromCaller, that would be nice to have a ignore param there

#

How can I get the ignore value from the CLI though 😮

spark cedar
#

instead of LoadContext, it should load the value from the value provided

fair ermine
#

Why doing so when the value can be ignored on first load?

#

If I do that, that means the dir will be load with all value, then copy with filters

spark cedar
#

because you might not always call a function with this from the cli

fair ermine
#

That's so true

spark cedar
#

you might call it from another module

fair ermine
#

Okay, hmm

spark cedar
#

but we still should have the cli optimization - that makes it nice and fast

#

but functionally, it's more important to have it in modfunc.go

fair ermine
#

So I need to copy the whole dir while applying the ignore pattern

spark cedar
#

yup 🙂

#

kinda sucks, but can't think of a better way to do it really

fair ermine
#

That's not nice, we cannot have a Without function that not take a path but also patterns?

spark cedar
#

we could maybe one day track if a directory has already been filtered with those filters? then we could avoid applying in twice

spark cedar
fair ermine
#

Yeah..

spark cedar
#

just a bunch of llb.Copys

fair ermine
#

Alright! I'll try to do that then

#

Using the same logic as loadContextualArg but with load Arg

spark cedar
#

if we hit real perf bottlenecks with this, and too much disk usage - there are ways to speed this up, but they involve a lot of internal work to do hardlinking of lots of things

#

essentially, we'd want a new FilterOp

#

similar to a discussed ChdirOp I've discussed with erik before

fair ermine
#

I see

bronze hollow
#

Can we have all environment variables prefixed with DAGGER passed to our functions? 😁

stuck bloom
#

Nice try! I wish the same 😄

wet mason
#

Can someone try, using the dev engine:

dagger --source . call sdk php generate export --path .
git status

I'm getting a formatting diff every time I try

/cc @stray heron @fair ermine @spark cedar @rancid turret

fair ermine
#

Strange

wet mason
#

I pinged the wrong Tom 🙂

wet mason
wild zephyr
fair ermine
bronze hollow
#

Can a Dagger module be spread across two files, or expose multiple objects?

wet mason
#

I nuked the engine volumes so I don't think it's cache related

#

Something in my .git?

still garnet
#

@wet mason is it any different if you use context dir instead of --source .?

#

maybe it's picking up ignored stuff?

wet mason
#

@still garnet it's a subtle thing where php code is formatted differently (like newline before brackets or something like that)

#

doesn't seem like it's grabbing more/less stuff

still garnet
#

no diff here fwiw

wet mason
#

@still garnet oh, got it -- I just wiped out sdk/php/vendor

#

it's .gitignore'd but I must have ended up in a broken state somehow

spark cedar
#

@tepid nova made some updates to the check function in our ci, so now we can easily check all the sdks with --targets sdk/, and even be more fine-grained with --targets sdk/go/lint.
lemme know what you think https://github.com/dagger/dagger/pull/8434

GitHub
ci: cleanup and extend checks api by jedevc · Pull Request #8434 · ...

Extended the checks api, so now we have sdk/go (and others), which contain sdk/go/lint, sdk/go/test and sdk/go/test-publish.
Also, I removed the --ref arg, this doesn't seem to be used anyw...

#

also, is there any way we could try passing in the commit id when we need it locally (instead of the whole id)? my .git dir locally is 150MB, which is pretty big to upload every time. that's part of the reason for passing in the version string right now
(also note, that the full git directory sometimes might not be in .git, especially with git worktree, etc)

#

happy to make the changes myself, just trying to understand the current state of things

wet mason
#

<@&946480760016207902> What's the weather forecast for a 13.1? 🙂

tepid nova
tepid nova
meager summit
bronze hollow
#

Sorry, tons of questions today. I'm trying to expose multiple services via dagger call XY up - but I don't think this is possible. Am i missing a trick?

tepid nova
#

At the moment you need to expose a proxy service that has dependencies to the other backend services. @tidal spire has a pretty good proxy module you can use

bronze hollow
#

I've tried that, the only service I can reach from the host is the proxy

#

Oh, you mean an actual proxy like nginx to forward

#

Eurgh

#

At-least it unblocks me. Thanks

tepid nova
#

@bronze hollow the thing is, one way or the other you will need a proxying layer since several services in your array might expose the same port number

#

One possible optimization could be to have the builtin tunnel flatten several services into one, and just give you an error if you have port conflicts

bronze hollow
#

I just modified all my modules to accept a port number argument 😅

tidal spire
bronze hollow
#

I am going with Devenv for local dev just now, there's too many pain points managing Dagger and then local processes.

Maybe something Dagger can handle in the future.

tidal spire
bronze hollow
#

It's neat but it doesn't help in this instance. I am trying to use Dagger for something it's not intended for, at least for now

#

I wanted to replace a Process / Docker Compose style setup

#

The proxy did work well but it's only half the battle.

#

I'm just eager. Dagger solves 8/10 problems ATM and there's that urge to take on the rest 😂

tepid nova
tepid nova
tepid nova
#

@fair ermine I have another context-directory fine-tuning request 🙂 Sub-modules should be excluded by default IMO. Will open an issue

#

@spark cedar the "directory hash" feature is cool for computing version info, but it means uploading the entire source directory even when I don't need it

#

just to compute that one hash

#

is there not another way?

tepid nova
tepid nova
#

Can anyone explain this snippet in engine test setup?

    engine := t.Dagger.Engine().
        WithConfig(`registry."registry:5000"`, `http = true`).
        WithConfig(`registry."privateregistry:5000"`, `http = true`).
        WithConfig(`registry."docker.io"`, `mirrors = ["mirror.gcr.io"]`).
        WithConfig(`grpc`, `address=["unix:///var/run/buildkit/buildkitd.sock", "tcp://0.0.0.0:1234"]`).
        WithArg(`network-name`, `dagger-dev`).
        WithArg(`network-cidr`, `10.88.0.0/16`).
        WithArg(`debugaddr`, `0.0.0.0:6060`)

Specifically I'm curious about why this extra config is needed for the engine & CLI tests, but not for the SDK tests. Those get an ephemeral engine too, but without all the custom config. Is that on purpose, or just an omission? Should I apply this extra config for the SDK tests too? Or will that break something important?

fair ermine
wet mason
#

I find it weird / frustrating that modules are not "self sufficient", e.g. we .gitignore codegen and overwrite it anyway even if there's a local copy

  • at least in Go it's an anti-pattern
  • opening a dagger pipeline blinks like a christmas tree in vscode unless you run dagger develop. Happens to me everytime the dagger src dir is moved around in our repo
  • what I see in my editor is not what I get. Code looks alright but then dagger call fails to compile
  • Breaks tooling
  • We waste time codegen'ing at every dagger call
spark cedar
# tepid nova is there not another way?

sadly, no, not until we can support lazy directory uploads (which we've discussed before, it's a pretty huge piece of work)
for version info, i've previously hashed the ID of the Directory, but also that still requires uploading it

#

contextual git would be useful here

fair ermine
bronze hollow
#

@fair ermine I'm trying to do more weird stuff and I was wondering if you have about 60 seconds ? 😄

#

I'll open an issue

bronze hollow
#

Obfusticating secrets within a terminal execution was a very nice touch I didn't expect

CLOUDFLARE_API_TOKEN=***

😄

fair ermine
spark cedar
#

what a fun feature, i don't think that's intentionally there 😄 just a feature of the way it's all architected

meager summit
#

hello folks, when adding a new field to a function with version specific Views, do we need to do anything specific for PHP SDK?

I added a new field to WithExec, but PHP check is failing.

still garnet
#

Can anyone explain this snippet in

#

I find it weird / frustrating that

tepid nova
spark cedar
#

getting a unique build id is common enough imo, if it hurts to do, we need to improve it - though outside of making everything lazy, i can't see how to do it very clearly

#

the entire source directory is going to get uploaded regardless of whether we call digest or not

#

if it's set as a contextual arg, doesn't it always get uploaded?

tepid nova
#

I just needed to figure out where it's needed. But I have the answer thank you! I'll tie it to engine build & test

#

ah one more: currently we use the hash for the CLI also. based on the above, it's not really needed for every build yeah?

spark cedar
#

it still is yes

#

dagger version prints out the semver version it was built with

#

we need to be very careful about changing logic around this fyi, there's a lot of stuff with version compat that depends very heavily on setting this correctly in the ci

tepid nova
#

right but if that version doesn't have a hash in it during development, will anything break?

spark cedar
#

having the right format of version string and the style of that is important - see version.go

tepid nova
#

but isn't the hash only used in development?