#🚀 v0.12.0 - 9th July 2024

1 messages · Page 1 of 1 (latest)

round geode
#

Hello all, kicking off this thread wayy early, since we have a proposed date, and quite a few things in flight before then 😄

#

So, the idea is we skip this and next week for releases, and aim for v0.12 the week after, to avoid clashes with 4th of July holiday.

#

main is open for breaking changes at this point I think - so we can start working through some of the drafts in the milestone. I think there's still potentially conversations around what's okay to break, etc, we should avoid breaking things in daggerverse as much as possible.

#

If we hit an urgent issue that means we have to release a v0.11.9 (hopefully not), the plan is to branch v0.11 off of v0.11.8, and cherry-pick any fixes to there.

#

cc @hard shoal @toxic finch @gloomy oasis @buoyant ledge @lean nest @noble basin @sullen zephyr @timber spade @lucid birch

toxic finch
#

So can I do the breaking change on the TS dependencies & drop commonJS support?

round geode
#

✨ v0.12.0 - 9th July 2024

#

As a non ts dev, that's up to you I guess! The thing to help decide is who this is going to break, and what their migration path will look like.
If it's a significant number who will need to take manual action, or the migration path is particularly tricky, it might be worth polling #typescript to see if they have opinions.

toxic finch
#

Okay, I'll tackle this problematic next week then, first I'm focus on context dir

noble basin
#

👍

sullen zephyr
#

What's the main motivation for waiting 2 weeks - marketing or engineering?

#

(not sure which, based on "avoid clashes with 4th of July holiday")

noble basin
#

2 weeks is an estimate of how long it will take us to get everything merged & ready for the release. The truth is that we can cut the release anytime, including the 4th of July. The date above is more of an estimate, we can adjust as needed.

This tracks what we think we want: https://github.com/dagger/dagger/milestone/45 . There are a bunch more things in flight here: https://github.com/dagger/dagger/pulls . Pretty sure there are changes coming which are in neither of these two places.

Are there things that you think we absolutely need to have in v0.12.0? If there is anything that you think we can defer post v0.12.0, that is likely to help move the release date closer.

round geode
#

most of the things in the milestone currently are breaking changes - those should be done in v0.12.0, or will have to wait till v0.13.0

sullen zephyr
#

I think the second we merged breaking changes into main, we accepted that anything in the milestone that is not yet ready, will have to wait for a follow-up

#

I don't like keeping main in a paralyzed unreleasable state in the middle of a period of frequent emergency fixes

round geode
#

this is the plan for emergency fixes

sullen zephyr
#

Are there things that you think we absolutely need to have in v0.12.0? If there is anything that you think we can defer post v0.12.0, that is likely to help move the release date closer.

I would love to have the luxury to answer that question, but it seems my hands are tied really

round geode
#

we can make more releases in the v0.11.X series, we just won't do so directly from main - this will avoid us accidentally pulling in huge changes that might also break legitimately

sullen zephyr
#

I don't think we've had to backport for releases in the past. Assume we will need to make at least one more .11 patch release. Are we comfortable actually doing this? The productivity tax is worth it?

round geode
#

i've done a lot of backported releases in the past with buildkit - overhead has generally been fairly minimal, i'm not too worried about it

#

the alternative seems to me that we merge all of our breaking changes within a tiny windows, and try and release immediately after to avoid closing main for too long, which means we have less time for internal testing

sullen zephyr
#

  • My marketing perspective is that it's better to release before July 4 than after, if possible, because that date coincides with the sharpest dropoff in engagement in our community. So it will be timed exactly for minimal impact

  • My engineering perspective is that even with a good plan B for patch releases, it's not healthy to keep main unreleasable for too long. So the sooner we get it out of the way, the better.

round geode
#

i'm not entirely convinced about being ready for the 4th - even if we drop everything but the breaking changes from the milestone, i think some of those deserve some careful thought to avoid breaking the daggerverse

sullen zephyr
#

To be clear I don't actually care what goes in the milestone.

#

So I'm not saying we should rush everything in the milestone

round geode
#

right, but we should try and avoid making breaking changes outside of minor releases

sullen zephyr
#

How many breaking changes are we talking about?

#

looks like a half-dozen roughly

round geode
#

everything with the kind/breaking label

sullen zephyr
round geode
#

we can definitely strip down the release and only do the kind/breaking changes

#

the reason for setting and coordinating a date is to try and give us time to get the release ready and in a relatively stable state after a large number of recent refactorings

#

not neccessarily for making a large marketing splash

#

at least from my point of view

sullen zephyr
#

OK. Then I'll leave it alone

#

My 2c, in the future maybe we should consider flipping it: merge breaking changes into a breaking-changes branch, and integrate there, then merge at the last minute for release. That way main is releasable at (almost) all times

round geode
#

that's kind of what we're doing, except they're all just individual prs

sullen zephyr
#

that seems to be a major difference, since we're measuring time to release those PRs in weeks

round geode
#

if we have a separate long-lived branch, now we need to manage that - what release does that go in? who tests that? who keeps it up-to-date with main?
i would strongly push back on having more than one long lived branch to work on

sullen zephyr
#

Yeah that's fair. I just think right now we're blind to the tradeoff this entails: we have mergeable PRs that are stuck in limbo for a long time, because they have a breaking change, and our process for managing those is "park all the breaking PRs until we are ready for a 3-week long stressful sprint to a minor release, with a not-completely-defined process"

round geode
#

if all of the prs are merged earlier, and the respective owners have the cycles to get those done sooner, we can release sooner than the suggested date

sullen zephyr
#

It kind of looks like we're holding on some of these PRs not because they're not ready, but because we're scared of merging a breaking change into main too soon

round geode
#

i mean, sort of, it's good to batch these changes together

#

if we just merge every breaking change as soon as it's ready, we lose user's trust, it makes it so much harder to convince users to upgrade to new versions

#

making upgrading a predictable and easy experience (even if that takes more engineering time) is definitely worth it in my experience
a badly executed release ends up with us attempting to recover for weeks after, our users frustrated, and everyone in critical bug-fixing mode.

sullen zephyr
#

That's fine with me

#

I'm just pointing out that you're already maintaining a virtual release branch

round geode
#

@hard shoal most of the api breaking changes are related to your work - do you feel confident in getting those changes in this week?

sullen zephyr
#

The sum of all PRs that you want to batch together as soon as the window opens for a breaking release.

#

I think you should consider frontloading some of that "batch it together" work to keep the release window for growing too large (3 weeks is a long time)

#

Also didn't we just merge a breaking change last week, thereby triggering this whole machinery without having any control over it?

round geode
#

i don't think the cadence of a release every 3 weeks is at unreasonable - we've settled into a cadence of 1-2 weeks recently, but we have done longer and we have done shorter
given that each of these is a reasonably large commitment of time (i'm working to automate more, but until then), i'm actually more convinced that a slightly longer period than we currently would be better

sullen zephyr
#

From my point of view:

  • Hey we just merged a breaking change
  • So we should probably cut a 0.12 release very soon. We can't make patch releases until we do
  • Also we should batch together as many breaking changes as possible
  • We have a half dozen of those pending, but it will be another 2 weeks to get them all ready to go
  • When would you like to release 0.12?

--> Does it matter when I want to release 0.12?

round geode
#

we have merged breaking changes between clients and engines before - the changes in the milestone potentially break engine<->module compatability - there's an impact on the daggerverse here, i don't think we should just rush into that

#

i need to head off for the day now, will pick this up tomorrow

timber spade
round geode
#

this changes an api served to modules

#

if older modules are connected to newer engines (like a depedency from the daggerverse), you will get stuck

timber spade
#

(Non urgent topic, please head off if needed)

round geode
timber spade
timber spade
round geode
round geode
timber spade
#

Yep, it’s a python 2->3 type situation

round geode
#

one day i think we should move towards a release branch setup, like containerd/buildkit/etc - i think something like that is super important for giving users reassurance about long term support, but we have too much in flux to think seriously about that right now, imo

sullen zephyr
#

To be clear this issue (module compatibility, version checks etc) is infinitely more important than a 0.12 release date.

#

I'm not sure if anything I'm saying about the release date might negatively impact module compat, but if it does, module compat wins

sullen zephyr
#

@timber spade looks like you already have a PR in that's worthy of a patch release -

timber spade
#

Yes, I'm catching up on the details in this thread, previously just read the part about incoming module-engine incompat.

Provided main in it's current state is releasable I'll go ahead and do that and setup a separate release thread.

sullen zephyr
#

Provided main in it's current state is releasable I'll go ahead and do that and setup a separate release thread.

I think part of the thread above is caused by the fact that main in its current state is not patch-releasable, because we have at least one breaking change. But don't quote me on that, I don't know for sure what the state is.

timber spade
toxic finch
#

Btw if there's anything you need a review on tonight ping me, I'm gonna work late

round geode
#

@ocean egret @dapper socket for daggerverse - since we expect to introduce breaking changes, do we have an easy way to "check" the published modules against a to-be-released version of dagger

#

i think the idea is to avoid breaking everything, but if we can catch things that we do, it would be great to get advance warning, and downstream any contributions, or maybe work out how to avoid making those changes in the first place

ocean egret
#

Would these breaking changes be caught by something like a dagger functions <module> call or similar?

round geode
round geode
#

if i'm allowed to ask for the moon, what would be even cooler would be a way to check dagger main against the daggerverse on a nightly basis, and be able to track and graph increased/decreased rates of failure

#

but obviously one-off would probably be fine here

ocean egret
#

Sounds doable! Would we only care about the latest version of each module or do we want to check against all versions? Or maybe we care to check only module versions that were built with dagger v0.x.y and above?

round geode
#

oof those are all good questions

#

all versions is "interesting", but not sure, i think there's likely to be a lot of drift, etc, at least at this stage in the ecosystem - might be worth trying? i think the latest is most important though

#

Built with v0.x.y is a good constraint to have, but we haven't broken any module compatability since we launched modules in v0.10.0

#

so we could have that as the lowest minimum for now

ocean egret
#

Sounds good. I'll get to it!

round geode
#

cheeeeers 🎉 should hopefully make our release (and all future releases) much smoother

dapper socket
#

let's go to lounge and I can explain how we're doing it

ocean egret
#

👀

sullen zephyr
#

To resurrect the topic of breaking changes and their relationship to releases: if we decided to be more strict about avoiding breaking changes, could we? And what price would we pay?

round geode
#

surfacing something from a dm with @hard shoal: tomorrow morning i'm going to look at trying to have dynamic schemas based on the engineVersion - so the breaking changes avoid breaking the daggerverse

round geode
# sullen zephyr To resurrect the topic of breaking changes and their relationship to releases: i...

this is possible - but generally we want to make breaking changes for things like APIs where the previous decision turns out to not be the right one.

#

there are other approaches - buildkit has never broken backwards compat
however, there are big costs to this:

  • innovation is restricted - every single change has to be made thinking about compatability, so you can't just "fix a bug", sometimes you need multiple fixes, for all the different variations of client/engine combos
  • codebase becomes much more complicated - all of these edge cases take handling, which both makes it harder to make simple changes, but also prevents outside contributions
#

personally, i think the "never break" is the right approach, but if we paid those costs, we'd be stuck with some weird api things indefinitely, and we'd slow engineering pace down - maybe one day, but i'd like to push that into the future until it's unavoidable

sullen zephyr
#

The part that I keep thinking about, is that a lot of these breaking changes are very superficial - changes in the graphql schema plus a little plumbing. If we could find a way to encapsulate these changes as different versions of an "internal module" - as opposed to special cases mixed everywhere in the code - then perhaps we could have our cake and eat it too?

#

That wouldn't make all breaking changes go away, but it would allow us to tier them in a 80/20 way maybe

round geode
#

If core could become a "proper" module, that's definitely doable

#

I guess we just gotta work out how to do it finally 😛

sullen zephyr
#

To summarize:

  • I 100% empathize with the idea that we still need freedom to break things - or progress will grind to a halt which could kill us
  • On the other hand, I also worry that too much breakage, too frequently, can create a "death by 1000 cuts" situation, that could also kill us. Especially when considering the secondary effects, like growing incentives to group breaking changes together (which trains users to not trust minor updates) and rushing breaking changes to keep the release window small (which increases the risk of quality problems)
round geode
hard shoal
#

Just a few notes on the impact of these PRs:

sullen zephyr
#

@hard shoal for 0.12, maybe we punt on Container.withNewFile? I worry that it will be the straw that breaks the camel's back. wdyt?

hard shoal
round geode
#

When would we punt it to?

#

If we don't change it now, for the reasons we want to, will we ever?

sullen zephyr
#

Can be temporarily replaced today with Container.newFile and Directory.withNewFile

I guess we have a one-time card we can play for non-deprecatable breaking changes: a one-time change of the API to remove the withXX style 😛 withExec becomes exec, etc. Then we would get a clean deprecation (at the expense of forcing absolutely everyone to change their code within the deprecation window). Not saying we should go for that nuclear option, but it is there.

round geode
#

I think its worth trying that approach if we can - we can get the best of both worlds - we remove the issue in new code, but don't break old module code

proven breach
#

So can I do the breaking change on the TS dependencies & drop commonJS support?
How are you planning on dropping support? This just in the code-gen side of things ?

ocean egret
round geode
#

That one should be good yes 🙂

ocean egret
# round geode That one should be good yes 🙂

So I already run the whole recrawl twice and in both cases I haven't seen any errors. I'm syncing with Marcos in a bit for a quick review to see if I'm doing things correctly.

This recrawl is only checking for the latest version of all modules. 566 modules were checked

#

Bundling this up into a dagger module that does the whole thing and putting that in a workflow dispatch would be fairly straighforward and could be an interesting community call demo party_gopher

ocean egret
#

I was looking for errors at the wrong table. We do have quite a few, so far we've found a few modules that are using the Terminal type which is marked as invalid type. @round geode I can send over the entire output string of the code generation command, or I can try to build something that processes this and tries to group by error. Which one do you prefer?

sullen zephyr
#

It may be way too late for that, but just in case: any chance we could morph the breaking Container.terminal() change into a non-breaking Container.terminal().attach()?

#

I don't see how we have to break the CLI behavior to get the new code behavior. We could just change the CLI code so that it calls the new attach() when receiving a Terminal return value

#

Would be one less headache

ocean egret
ocean egret
# round geode That one should be good yes 🙂

First file contains a summary created by ChatGPT that is supposed to clusterize the errors, haven't checked for each one individually yet and it seems to have duplicated the Terminal error, but it might be a good enough proxy. The second is a deduplicated output that for contains the error encountered for each module+ref. I validated each of those failures using a dagger functions call as well and I can see that it fails with the same error

📎 Error_Cluster_Summary.csv 📎 deduplicated_output.csv
timber spade
#

Yeah this has been on my brain's backburner for a while. The biggest trickiness is that everything I can think of would still result in some API that may end up with backwards incompatible changes. i.e. if core is a true module then it still talks to the engine somehow and that API can break.

It still is most likely worth doing since I'd rather deal with breaking changes to an internal-only API than a user-facing one, but I'm not sure yet whether it actually ends up reducing our maintainance burden overall, which also the same primary source of pain for dealing with version compat across breaking changes in core today.

It's just all hard to say since everything is still fuzzy+hypothetical. Agree that once I experiment with moving off LLB we'll be able to get a better concrete idea on what this would look like in practice.

sullen zephyr
#

100% to be clear I'm not talking about fully swappable core modules with a stable public API (which would just shift the problem as you mentioned), I'm picturing more a private, go-only API, something just barely above llb/solve. An architecture that would give us some breathing room by making things more modular internally

sullen zephyr
round geode
#

so we should have this as a changelog However, it's weird that the *Terminal type is now gone.

sullen zephyr
#

if doable we could get that Terminal type back, no breaking change, get our cake and eat it too

round geode
#

Sorry, missed this - I think the ergonomics of the new Terminal make a lot of sense - I would rather users didn't have to add attach into their code as well.

#

I'm gonna investigate the dynamic module api thing, which is on my todo list today - maybe we can pull that trick here too

#

Another idea I had was that we could alias Terminal to Container

round geode
round geode
#

one thing i have thought about here - having dev-... versions in engineVersion really hurts, because we can't compare version numbers elegantly - thankfully, i think this is a bit of an edge case, users are installing from main or dev versions already there, so that's already a bit of no-mans land
at some point, we should switch to go-style pseudo-versions (https://github.com/dagger/dagger/issues/7760)

round geode
hard shoal
round geode
#

ahh sorry! that was gonna cause a lot of conflicts regardless sadly, but at least now it looks like tests are running a bit faster 🎉

hard shoal
#

Totally fine, I was looking forward to trying out those changes and it needed to be done 🙂

round geode
#

turns out this is doable, but with much pain 😢 i've ended up having to introduce a new dagql concept of "views", which allow multiple different perspectives on a single server.
without this, you end up with a server for each version of the schema, which means that ids valid in one schema might not be valid in another schema (and so isn't a viable approach)

#

anyways, i remain very hopeful, it's just... taking time to hack through

toxic finch
round geode
#

Also, we need to make sure that this doesn't break any ts modules that work today in the daggerverse

toxic finch
#

Ohhh yeah, I need to check that

lean nest
#

hey everyone! Gerhard will be ping the owners of certain PRs soon to add details into the blog post draft. I apologize for the short notice, but please add your content to the draft tomorrow. Once you add the content, Neela will clean it up.

This will give Solomon has enough time to review on Wednesday before the US holiday and PTOs, so we can get it out shortly after the release. Thanks all!

round geode
#

technically it should be readable enough if anyone is curious to review, but going to tidy up commit history tomorrow to make it a bit easier (i also want to rebase-merge this one)

round geode
toxic finch
#

Yup, I’m on it, normally nothing on the user end, it just drops support for commonjs but we set it in the runtime

#

Most modern js is compatible with ES2016 so should be good

#

I’m still checking and trying some local test to catch any issue

sullen zephyr
#

Hello hello. Given the unfinished design discussions that we want to wrap up before release:

  1. Container autoprint, and autoprint in general
  2. TUI "lingering"
  3. Traces integration into TUI
  4. Breaking change to export string argument?

Should we start planning for a possible delay in the release?

I'm available today (on phone while traveling) and tomorrow (fully) on west european business hours, to participate in the discussion.

But it seems optimistic that everything is 100% wrapped up across many TZs, and any changes we agree to are shipped and confidently tested by tuesday morning Pacific.

wdyt?

round geode
#

apparently you're good at reading minds 😄

#

i was gonna ask this exact thing at the end of today, as us folks came online

#

but yes, i absolutely think we should be delaying - the milestone has 15 open items on it

hard shoal
#

Yeah, agree on delaying the release too.

toxic finch
#

Yup agree too!

#

@round geode Did we have any issue with the Go CI, I keep hitting an issue with:

dagger call -m dev --source=.:default sdk go test

--- FAIL: ExampleGitRepository (2.24s)
panic: input: git.tag.tree.file resolve: failed to update submodules for https://github.com/dagger/dagger: git error: exit status 128
stderr:
fatal: No url found for submodule path 'engine/telemetry/opentelemetry-proto' in .gitmodules

 [recovered]
        panic: input: git.tag.tree.file resolve: failed to update submodules for https://github.com/dagger/dagger: git error: exit status 128
stderr:
fatal: No url found for submodule path 'engine/telemetry/opentelemetry-proto' in .gitmodules

I can reproduce it locally, but it's also happening in GH: https://github.com/dagger/dagger/actions/runs/9839011704/job/27160149479?pr=7794

Java and Rust are also breaking even if I didn't touch the code in the PR (https://github.com/dagger/dagger/pull/7794/files): https://github.com/dagger/dagger/actions/runs/9839011704/job/27160149690?pr=7794
https://github.com/dagger/dagger/actions/runs/9839011704/job/27160150335?pr=7794

Are we aware of this issue?

#

I also see the issue on main btw

sullen zephyr
#

Container autoprint, and autoprint in general

I am focusing on that one at the moment

round geode
#

hm i think this is actually potentially a weird buildkit issue i've seen before - if you clear your cache, does it start working again?

toxic finch
#

I mean it fails on the CI & locally

#

So I'm not sure cache is the case, but yeah locally I start from a clean engine

#

It also make test all & testdev failing

round geode
#

what, that's so odd

#

i didn't see this in my pr

toxic finch
#

Check mainjob

#

It's because you deleted the .gitmodules in your PR

#

Looks like it match the error I see in tests

round geode
#

dammit, i only half removed the submodule

#

hate submodules, genuinely full of misery and nastiness

toxic finch
#

No problem party_blob I approve your PR, merge it whenever you want

round geode
#

hm, still looks like something is wrong

#

will follow-up

toxic finch
#

Merge it whenever you want btw

toxic finch
#

Oh I guess I need to regenerate the doc

round geode
#

hm i might have missed that one

#

serves me right for trying to balance 100 prs at a time

toxic finch
#

Yeah I have the same issue haha

round geode
round geode
#

@hard shoal do you need a hand on adapting the breaking changes with the views/etc? happy to lend a hand on those if needed

#

if anyone has a moment to think about https://github.com/dagger/dagger/issues/7760 that would also be wonderful to include in this release - but there's some very fiddly subtlety about ordering, e.g. the practice of comparing to not-yet released versions is probably a bit confusing in this case.

#

I'm very split on what to do with https://github.com/dagger/dagger/issues/6894 - I would like to implement the fix I suggested with a fancier understanding of max-parallelism to handle nesting, but this it sounds like a "medium t-shirt" task potentially. Removing the option is a "x-small", so could be done very easily.

timber spade
noble basin
timber spade
sullen zephyr
timber spade
craggy osprey
timber spade
timber spade
lucid birch
#

@timber spade re: "legacy" ... not sure where we landed on allowing non-module code to invoke modules?

#

(not urgent convo, we can postpone this thread until after the release)

#

@timber spade basically wondering if it could be a path to test dagger modules using native frameworks if e.g. we have bindings to invoke

timber spade
lucid birch
timber spade
# lucid birch <@949034677610643507> basically wondering if it could be a path to test dagger m...

That's a route, yes, I'd be more in favor of something that's sort of a hybrid where we just make it easy to wrap external frameworks by implementing a small module and then still invoke it with dagger, sandboxed, etc. https://github.com/dagger/dagger/issues/6724

Supporting native frameworks is the way to go without a doubt; we're not gonna write custom test frameworks for every SDK language obviously. But just supporting i.e. go test directly without module wrapping reverts us back to requiring certain host dependencies at the right version, etc. (which gets especially painful for languages like Python/TS/etc.), which I'd really like to avoid.

#

So I think it's worth finding some path that lets you e.g. write _test.go files the same you normally would, but then you still invoke the tests using dagger. Same idea for other test frameworks in other languages.

And then to solve the problem w/ there being many different test frameworks for languages like python/ts, we make the support for a given test framework implemented as module (one idea in that issue is a module function that accepts another module as an argument, which is possible today).

Hazy-ish idea still but I think there's something there.

lucid birch
#

Even if it's neatly wrapped behind a e.g. dagger.Connect() that returns a codegen'd dag? In which case it would be very similar to invoking modules from within a module

#

Thinking about the invocation part wrapped by dagger ... it's somewhat "lossy" since we'd lose that part of native (e.g. for instance, in Go you can click to run one test from vscode, in TS when using jest the tool can do live watch and re-test, etc etc)

timber spade
timber spade
lucid birch
#

Oh yeah. Looks like it’s on the host but it’s not. Similar as to when you invoke a dependent module

#

dag.MyModule would behave like dag.OtherImportedModule — just codegen’d

#

Just a convenience to pilot the module for testing using native local code, but it’s as if you were issuing dagger calls

timber spade
#

I think it would be different than https://github.com/dagger/dagger/issues/5993 then. For that, we just wanted to make modules invokable via a library in the same way "legacy" SDKs enable you to invoke core APIs from a library. I don't think we could support containerizing that in general (the code might just be a compiled library inside some binary that's now executing on the host, can't really just teleport it to the engine in a container in the general case).

timber spade
# lucid birch Just a convenience to pilot the module for testing using native local code, but ...

I see, yeah I think that would be nice to have for the IDE integration use case you mentioned. I can see how we'd have dag basically just switch behavior depending on if it's already in a dagger container or not (if not in container, ship the test code to the engine and invoke the test there; otherwise just run the test). But I think it would be highly specific to this "test" use-case (it requires the source code is available, etc.)

round geode
round geode
#

does anyone needs reviews, help, etc with anything on the milestone? happy to trade, i've got a couple prs that could use some eyes 👀

#

@toxic finch do we need a typescript issue in the milestone for the current deps issue etc you're working on?

toxic finch
#

Hmm no I think it's okay, I'll finish the package manager config tomorrow

round geode
#

Awesome awesome 😎

hard shoal
#

@toxic finch, there's an issue with a TypeScript test. It's failing in main:

dagger call -m dev --source=.:default test custom --pkg=./core/integration --run="Module/TypescriptRuntimeDetection"

https://dagger.cloud/dagger/traces/f6e7a847b5fdff1b14b0c3dccf91be05?span=a3d0d1b65eb276af


    Error Trace:    /app/core/integration/module_typescript_test.go:616
                                /app/testctx/testctx.go:163
    Error:          Not equal: 
                    expected: map[string]interface {}{"runtimeDetection":map[string]interface {}{"version":"node@20.15.0"}}
                    actual  : map[string]interface {}{"runtimeDetection":map[string]interface {}{"version":"node@20.15.1"}}
                    
                    Diff:
                    --- Expected
                    +++ Actual
                    @@ -2,3 +2,3 @@
                      (string) (len=16) "runtimeDetection": (map[string]interface {}) (len=1) {
                    -  (string) (len=7) "version": (string) (len=12) "node@20.15.0"
                    +  (string) (len=7) "version": (string) (len=12) "node@20.15.1"
                      }
    Test:           TestModule/TestTypescriptRuntimeDetection/should_detect_pinned_lts_node_version
toxic finch
#

Ohhh yeah, okay I understand why

#

Will do a fix

toxic finch
# hard shoal <@281874480651829250>, there's an issue with a TypeScript test. It's failing in ...

Hey, I fixed the tests, lts has change its image and the extra logic needed to accept a sha256 would be pretty tricky for because we need an alpine image for the runtime.
I tried few things but it wasn't good enough IMO:

  • Reparse the version to see if a @ is there, meaning we need to add the extra sha after -alpine
  • Check if alpine is already present, then ignore it.

However, this hide a lot of things to the user, I think he expects to just set a version, not an actual docker image version.
I want to get your thought on that though, I can make the changes if you think that's okay.

https://github.com/dagger/dagger/pull/7861

GitHub
fix(ts-sdk): test detect node-version by TomChv · Pull Request #786...
hard shoal
round geode
sullen zephyr
#

FYI I'll be on a podcast recording from 4pm to 6pm FR (3pm-5pm UK, 7am-9am SF). Then available again for release support.

#

@round geode @hard shoal yesterday Justin said I should check with both of you on the status of breaking changes in the milestone - what to do about them; whether we should consider dropping some from the release; and what would be consequences if we did.

round geode
round geode
hard shoal
round geode
#

WithNewFile is a similar hell i imagine

round geode
hard shoal
gloomy oasis
#

Feel free to ping me with any release blockers (i.e. reviews), my current task isn't one

round geode
#

@gloomy oasis ^

hard shoal
round geode
round geode
sullen zephyr
#

back

round geode
# sullen zephyr back

good news 🙂 i have a clever idea for making sure that dev versions in engineVersion don't make life absolutely miserable 🙂

#

as requested salute

round geode
#

In terms of timing for tomorrow - I'm probably looking at kicking off the release towards the evening my time, so that there's a day to finish off anything last minute, do reviews, run any last minute manual tests, etc.

#

Slightly after the community call probably

sullen zephyr
#

@round geode does that leave enough time for actually releasing on thursday?

round geode
#

Yeah, I'll run into my evening 🙂

#

If it all goes to plan, I'll head off earlier than usual on Friday 🙂

sullen zephyr
#

Maybe a good opportunity to test a EU->US release handoff?

timber spade
#

I'm happy to pick up too

#

Yeah what Solomon said haha

round geode
#

Oh yeah of course 🙂 sorry, that slipped my mind 🙂

#

Let's go with that then, I'll try and get everything into a good-to-tag state

#

And @timber spade will pick up from there

timber spade
#

Unless I should wait

round geode
round geode
# timber spade I'll look now

Happy to have a review in-principle 🙂 but the code is definitely broken right now for reason I don't understand (first commit should be good though)

sullen zephyr
#

@timber spade I just ran into the mother of all examples for the internal/external call discrepancy, 15mn after we discussed it:

  • Parallelize a function by using goroutines/errgroups: now it needs a context
  • Function is super low-level, dozens of direct callers, need to pass the context from all of those
  • Half of the callers didn't take a context, repeat the cycle with hundreds of their own callers
#

Actually in this case it may not be worth plumbing a context all the way down

timber spade
#

Just checking if this is fixed by self calls ^

To some extent, this is just Go being Go and something you always have to deal with (the perfect use case for AI tools, completely mindless+tedious changes that are hard to automate). But self calls would help a bit by enforcing consistency through codegen

sullen zephyr
sullen zephyr
round geode
sullen zephyr
#

I also hit an issue where the terminal stays open for a few seconds, then goes away. Worked fine while it was there. Happened consistently too. Anyone else got that?

hard shoal
sullen zephyr
gloomy oasis
round geode
#

yeah, there's still some bigger pieces that still aren't in 😱

sullen zephyr
#

So, what's the verdict on pulling the trigger today?

round geode
#

since tom is on holiday today

#

i'm kinda tempted to not take that one - since it potentially breaks users, and tom isn't around to help sort that out if it goes wrong

sullen zephyr
#

If you're not sure about taking it, don't take it

#

The whole theme of this release is quality, definitely not the time to squander it with a rushed timebomb

round geode
sullen zephyr
hard shoal
round geode
#

also a fun thing i've worked out we can do for next time - we'll be able to merge+test breaking changes for any future release we like 😄 and keep it disabled for normal users

#

so we won't need to keep these prs around until the very last second

hard shoal
sullen zephyr
round geode
#

yup - all you'll need to do is put a view for something like "0.13.0" - and it'll be hidden for everyone 🎉

craggy osprey
#

Realize we need to update the dagger call -m help text. Which is preferred?

The example below changes the current one:

  1. s/github repo/git repo/
  2. drops the local path example (e.g. "/path/to/some/dir")
  3. uses a real working example module ref
  4. shows the same format as the Daggerverse publish page (USERNAME/ORG etc bikeshed possible)
  -m, --mod string      Path to the module directory containing the dagger.json config file.
                        Either local path or git repo:
                        GITSERVER/USERNAME/REPOSITORY[/SUBPATH][@VERSION] (e.g.
                        "github.com/dagger/dagger/dev@main")

@sullen zephyr or others got a preferred set of these?

Another nice one is prob 1, 3

round geode
#

I think just 1 is good

#

if we do 3, we should use a canonical hello-world example

#

our own ci is probably not the right example here, it's definitely a deep end to throw users in

sullen zephyr
#

definitely use hello (eg. github.com/shykes/hello or equivalent) and not our own repo, since it happens to be in flux /dev, no /dev, etc

craggy osprey
#

Yep, this is the current state

  -m, --mod string      Path to dagger.json config file for the module or a directory
                        containing that file. Either local path (e.g. "/path/to/some/dir") or
                        a github repo (e.g. "github.com/dagger/dagger/path/to/some/subdir")
round geode
craggy osprey
#

Okay, something like this look nice @round geode ?

$ dagger help call
...
-m, --mod string      Path to the module directory containing the dagger.json config file.
                        Either local path or git repo

$ dagger help publish
...
  -m, --mod string   Path to the module directory containing the dagger.json config file
round geode
#

i think we should keep the reference to a git repo for now

#

oh it is there

#

sorry, it's been a day

#

yeah that lgtm

craggy osprey
#

It's there for call, install
Was testing for publish where we substring it

sullen zephyr
#

@round geode 3 questions:

  1. Are we still releasing today
  2. How can I help
  3. Are we still handing off to the US so that you can go to bed at a normal hour
round geode
#

we have one more from myself, and another couple from @hard shoal

#

if we do it, i'll definitely be handing off

sullen zephyr
#

Can the US start a release?

round geode
#

yup for sure 🎉

#

i'm happy to keep hacking on my bits for a bit more, but if @hard shoal has to head off, we should probably hold a bit more

#

i'm really not a fan of releasing tomorrow, but potentially we could do so monday

hard shoal
timber spade
#

I've been doing a lot of "laying without actually falling asleep" recently for some reason so I've been online later than usual, but can be here at like 8am PT to pick up

sullen zephyr
#

So today is definitely off the table?

#

laying without actually falling asleep

The worst

timber spade
#

I'm reviewing @jed's PR again now. Provided that and the other remaining things are merged by like say 1pm our time I think we can do the release today?

#

I just need to avoid it going too late since I have some IRL deadlines tonight (picking houses to tour)

round geode
#

@hard shoal can we group the docs updates into the v0.12 milestone?

hard shoal
round geode
timber spade
round geode
#

100% - i will be sleeping by then 😄

#

or trying to 😄

sullen zephyr
#

anything I can do to help ?

round geode
round geode
timber spade
#

@jed do you think it's crucial to get https://github.com/dagger/dagger/pull/7858 in for v0.12.0 or can we merge it right after? Just thinking about how it touches on parts related to CI quite a bit, which is always a risk

sullen zephyr
#

I just realized something... we removed Go aliases right?

#

that's going in 0.12?

round geode
#

yes it is

sullen zephyr
#

OK then we need to plan a major developer relations effort, to get people to migrate their modules

round geode
sullen zephyr
#

We bought ourselves some time with compat mode obviously. But they still need to port their code, or they effectively aren't really upgrded

#

Which they will soon realize when eg. they try to call the cool new Terminal() etc

#

Then what happens

#

-> we need to have a good answer to that question

#

cc @lean nest @craggy osprey @high ember

timber spade
sullen zephyr
#

As a start: all of us need to update our own modules, and report back on the experience

#

One more huge thank you @round geode (and everyone else who helped) for this compat mode - without it, we basically couldn't release at all as is (would be blocked not just on planning this devrel effort, but actually executing it in parallel)

hard shoal
#

Compat mode is all @round geode, just been cheerleeding him on!

timber spade
#

^ Yeah I was mostly just paranoid about this months ago and added the groundwork for it but Justin did all the actual implementation to get it working

hard shoal
craggy osprey
gloomy oasis
#

maybe gofmt could cover it?

hard shoal
#

There's an escape hatch to reduce work if needed through "star" import.

craggy osprey
#

Only Go SDK affected, right?

gloomy oasis
#

does this work?: gofmt -l -w -r 'Container -> dagger.Container'

craggy osprey
#

Container -> dagger.Con... oh come on! That 👆 😉

hard shoal
gloomy oasis
#

seems like it works, we just need one for every type 😛

round geode
#

@gloomy oasis also had the good idea of pointing users to . imports

hard shoal
#

Get the list from dag.TypeDefs(), daggerize it! 😛

round geode
#

fyi - this is the kind of info we should be capturing in our release notes

craggy osprey
#

we need the new import line too, right?

round geode
#

a bit late now - but actually, if anyone has a moment - going through all the .changes breaking changes and adding migration instructions as a multi-line part of the body would be super useful

craggy osprey
gloomy oasis
# round geode <@108011715077091328> also had the good idea of pointing users to `.` imports

doesn't work due to conflicts 😦

./dagger.gen.go:25:6: Tracer already declared through dot-import of package dagger ("concourse/internal/dagger")
        ./internal/dagger/dagger.gen.go:27:6: other declaration of Tracer
./dagger.gen.go:38:6: DaggerObject already declared through dot-import of package dagger ("concourse/internal/dagger")
        ./internal/dagger/dagger.gen.go:52:6: other declaration of DaggerObject
./dagger.gen.go:40:6: ExecError already declared through dot-import of package dagger ("concourse/internal/dagger")
        ./internal/dagger/dagger.gen.go:96:6: other declaration of ExecError

but the idea was to just add . "mymod/internal/dagger" to the imports

round geode
#

grr

#

ExecError can technically be moved

#

moving DaggerObject is a pain

#

I tried for that one, it's so linked into the go codegen it was not particularly simple

gloomy oasis
#

gofmt works but there are a lot of things to fix since it also includes e.g. ContainerWithExecOpts

craggy osprey
gloomy oasis
#

yep, using Go's official formatting engine, which is great for code mods like this, except we have to tell it each thing to fix by hand

sullen zephyr
#

Didn't we completely forget to mention this as a breaking change in the release post?

#

It's probably our biggest breaking change

#

@lean nest 👆

gloomy oasis
#

i think i can script this based on go build error output, 1 sec possibly many secs

round geode
#

i was gonna say

#

how good are you

round geode
#

sorry, just realized - atm, we've never made a big breaking changes to modules - so we can assume that any non-semver engineVersion is compatible with v0.11.9
if we publish v0.12.0 without this, we end up in a state where dev versions after v0.12.0 aren't compatible like this

#

we could just very nicely tell everyone to not use engineVersion with dev versions like that 🤔

#

mayyybe it's okay

timber spade
gloomy oasis
#

crap, found a flaw in my dastardly plan

image.png
timber spade
round geode
#

fyi the changes to dev/mage is kind of related to the core dev chat yesterday

#

instead of doing two dagger calls for the cli+engine, we can just do one

gloomy oasis
#

might have found a bug:

⬢ [fedora-toolbox:38] ❯ dagger-dev develop
✔ connect 0.1s
✔ moduleSource(refString: "."): ModuleSource! 0.0s
✔ ModuleSource.kind: ModuleSourceKind! 0.0s
✔ ModuleSource.resolveContextPathFromCaller: String! 0.0s
✔ ModuleSource.resolveFromCaller: ModuleSource! 0.0s
✔ ModuleSource.asModule: Module! 0.1s
✔ Module.sdk: String! 0.0s
✔ ModuleSource.resolveFromCaller: ModuleSource! 0.0s
✔ ModuleSource.sourceSubpath: String! 0.0s
✔ ModuleSource.resolveFromCaller: ModuleSource! 0.0s
✔ ModuleSource.asModule(engineVersion: "latest"): Module! 0.1s
✔ Module.generatedContextDiff: Directory! 0.0s
✔ Directory.export(path: "/home/vito/src/daggerverse"): String! 0.0s

daggerverse/concourse on  main [$!?] via 🐹 v1.22.1
⬢ [fedora-toolbox:38] ❯ go build .
# concourse
./dagger.gen.go:110:13: undefined: JSON
./dagger.gen.go:124:13: undefined: JSON
./dagger.gen.go:200:9: undefined: JSON
./dagger.gen.go:210:9: undefined: JSON
./dagger.gen.go:248:12: undefined: JSON
./dagger.gen.go:260:12: undefined: JSON
./dagger.gen.go:277:12: undefined: JSON
./dagger.gen.go:289:12: undefined: JSON

are we somehow not qualifying references to scalar types in generated code for dagger.gen.go?

#

here's my somewhat hacky script if anyone wants to try it:

set -e -u -x

sources="$(ls *.go | grep -v dagger.gen.go)"

while true; do
  go build . 2>&1 | grep undefined: | awk '{print $NF}' | sort | uniq > /tmp/undefined
  if [ ! -s /tmp/undefined ]; then
    echo "done"
    break
  fi
  echo "fixing $(wc -l /tmp/undefined) undefined symbols: $(cat /tmp/undefined | xargs)"
  for x in $(cat /tmp/undefined); do
    gofmt -l -w -r "$x -> dagger.${x}" $sources
    # NOTE: on Mac you might need to change this to `-i ''`
    sed -i -e "s/dagger.${x}:/${x}:/" $sources # gofmt seems to mistakenly replace field usage, but not definition
  done
  goimports -w $sources
done
#

which currently goes into a tailspin due to the above issue, but otherwise fixes my code

round geode
#

potentially

#

do you know which instance of JSON it is? I thought I got all of them

gloomy oasis
#

they're all fields in MarshalJSON/UnmarshalJSON

image.png
round geode
#

interestiiing

#

which specific module is this sorry, i have an idea for this

gloomy oasis
timber spade
gloomy oasis
#

I think it might be line 420 specifically - since JSON is a type alias

round geode
#

it could also be:

diff --git a/cmd/codegen/generator/go/templates/module_objects.go b/cmd/codegen/generator/go/templates/module_objects.go
index f4a91c0dd..b4d2286b2 100644
--- a/cmd/codegen/generator/go/templates/module_objects.go
+++ b/cmd/codegen/generator/go/templates/module_objects.go
@@ -444,7 +444,7 @@ func (spec *parsedObjectType) concreteFieldTypeCode(typeSpec ParsedType) (*State
         s.Id(typeName(typeSpec))
 
     case *parsedIfaceTypeReference:
-        s.Op("*").Id(formatIfaceImplName(typeSpec.name))
+        s.Op("*").Id(formatIfaceImplName(typeName(typeSpec)))
 
     default:
         return nil, fmt.Errorf("unsupported concrete field type %T", typeSpec)
gloomy oasis
#

i'll try things out and put up a PR

round geode
#

nope, that's definitely not it

gloomy oasis
#

well, this works, but i don't know if it breaks other things - looking for a way to deduce whether the qualifier is needed

diff --git a/cmd/codegen/generator/go/templates/module_objects.go b/cmd/codegen/generator/go/templates/module_objects.go
index f4a91c0dd..795c23470 100644
--- a/cmd/codegen/generator/go/templates/module_objects.go
+++ b/cmd/codegen/generator/go/templates/module_objects.go
@@ -417,7 +417,7 @@ func (spec *parsedObjectType) concreteFieldTypeCode(typeSpec ParsedType) (*State
                        s.Op("*")
                }
                if typeSpec.alias != "" {
-                       s.Id(typeSpec.alias)
+                       s.Id("dagger." + typeSpec.alias)
                } else {
                        tp := typeSpec.GoType()
                        if basic, ok := tp.(*types.Basic); ok {
#

i'm assuming that would break with locally-defined aliases?

#

maybe parsedPrimitiveType needs a moduleName field too

round geode
#

ohhh i think i see what's happening - it's seeing this as an alias type - but the alias type is being removed

#

is go somehow analyzing the dagger.gen.go?

#

i think it must be

round geode
hard shoal
#

Damn, I'm having "no space left on device" issues

timber spade
hard shoal
#

Yes

round geode
#

i think this is the 75% issue

#

i've been having this too, my disk without dagger is about 50% full

#

so there's no way dagger can have 75%

gloomy oasis
#

have you tried rm -rfing your home directory? worked for me

hard shoal
#

I already deleted more than 100GB but something's gobbling up my disk. Is there an easy way to see which processes are consuming disk space in macos?

round geode
gloomy oasis
hard shoal
#

Jesus, problem seems to have gone away after closing Safari 😮

#

And closing orbstack. But already closed orbstack and didn't fix.

#

71GB for dagger-engine.dev

timber spade
gloomy oasis
#

what does this mean again?

✔ connect 0.1s
✘ initialize 2.7s
! get module name: invalid character 'i' in literal false (expecting 'l')
  ✔ resolving module ref 0.1s
  ✔ installing module 2.6s
  ✘ analyzing module 0.0s
  ! get module name: invalid character 'i' in literal false (expecting 'l')

(something about parsing 'failed' instead of 'false'? but not sure where)

round geode
#

there will be "something" in the logs

gloomy oasis
# 
time="2024-07-11T19:08:03Z" level=error msg="failed to serve request" client_hostname=dev.fedora client_id=x5f2gntynfc29xadebh2oz3kd error="failed to get schema: failed to get schema for module \"concourse\": failed to create function \"resource\": failed to find mod type for function \"resource\" arg \"source\" type" session_id=zqvcnhamyhwhefldv2h6e526v
round geode
gloomy oasis
#

hmm, seems unhappy with the JSON field, even though it compiles

round geode
#

remember graphql indicates failure not through http status codes 🙂

timber spade
craggy osprey
round geode
gloomy oasis
#

will push one now

#

kinda surprised by the error tbh, though I haven't tried to run this module in a while. seems unrelated to the original issue

timber spade
round geode
#

yeah, i'm actually gonna cherry-pick the important changes over

#

the main change in here is that it affects dev versions

#

which should mostly affect us - i think we can merge after the release, and let it soak on main for some time

timber spade
gloomy oasis
#

but then at runtime it seems unable to find the JSON type

timber spade
#

Ah okay. Eh I should just fix that in case it gets hit by users somehow

gloomy oasis
#

lemme try this with v0.11.9 to see if it's not a new regression

#

yeah fails with v0.11.8 too (cc @round geode - so I think that PR might be OK as-is, except I should add a test)

round geode
#

we should definitely have some tests for the JSON case yeah

#

i don't think we do anywhere

gloomy oasis
#

it's possible the test just won't pass anyway because of the existing breakage, and this entire issue is kind of a nothingburger for v0.12 as a result. writing it up anyway, we'll see, maybe i can find the original breakage too

#

yep

        ✘ TestGoJSONField 20.6s
        ! test failed
        ┃
        ┃         Error Trace:    /app/core/integration/module_test.go:1400
        ┃         Error:          Received unexpected error:
        ┃                         input: container.from.withMountedFile.withWorkdir.withExec.withNewFile.withExec.stdout resolve: process "dagger --debug query" did not complete successfully: exit code: 1
        ┃
        ┃                         Stderr:
        ┃                         Error: make request: invalid character 'i' in literal false (expecting 'l')
        ┃         Test:           TestModule/TestGoJSONField
          ✔ /.dagger-cli session --label dagger.io/sdk.name:go --label dagger.io/sdk.version:n/a 20.6s
hard shoal
gloomy oasis
round geode
#

ohhhh

#

is this a capitalization issue 😢

hard shoal
gloomy oasis
# round geode is this a capitalization issue 😢

possibly - looking in the logs from earlier, I see:

❯ docker logs -f dagger-engine.dev 2>&1 | grep 'Json'
time="2024-07-11T19:08:03Z" level=debug msg="module did not find scalar" mod=concourse scalar=Json
time="2024-07-11T19:41:46Z" level=debug msg="module did not find scalar" mod=concourse scalar=Json

which could be the root cause for the module being unable to find the type, but I'm not sure why it wouldn't have failed earlier

#

oh, maybe that's tied to the same lookup action that returned the error

#

is it weird that there's a TypeDef for Json in the first place? as opposed to treating it like a built-in?

#

@round geode fwiw I don't think we need to worry about this for v0.12, since it's just as broken as before, so don't hang around past 9pm for this unless you have nothing better to do 😛

round geode
#

yeaaaaaa we can ship this bug 😄

gloomy oasis
#

will merge once CI is ✅

round geode
#

niiiiice 🎉

#

wait really? that's it?

#

sigh we love acronyms

gloomy oasis
#

yep lol

round geode
#

is this familiar to anyone?

#

i really don't get this one, it just simply "does not seem possible"

gloomy oasis
#

saw that one just today too, in another PR. I think it's a new test?

round geode
#

it's a new test, yeah, but the issue seems to be coming from go codegen

#

i guess this looks similar to the weird python equivalent where we get "main module not found"

#

or no, this error comes from the cli

timber spade
timber spade
round geode
#

my worry is that somehow this would have appeared when introducing views (which is also when these tests appeared)

#

however, views shouldn't affect presentation of non-core modules

#

so i'm just kinda confused 😄

#

if it does help, i have seen this issue locally @timber spade

#

huh, i just tried running all of TestLegacy, and I got something even weirder:

                Error Trace:    /app/core/integration/legacy_test.go:295
                Error:          Received unexpected error:
                                input: container.from.withMountedFile.withWorkdir.withExec.withNewFile.withNewFile.withExec.stdout resolve: process "dagger --debug call skip stdout" did not complete successfully: exit code: 1
                                
                                Stderr:
                                1   : 2fedb8a31a0e3ca7: initialize
                                2   : 63945c60b2c885cf:   resolving module ref
                                2   : 63945c60b2c885cf:   resolving module ref DONE [0.1s]
                                3   : b09987408c78c01d:   installing module
                                3   : b09987408c78c01d:   installing module ERROR [3.1s]
                                3   : b09987408c78c01d:   ! input: module.withSource.initialize resolve: failed to initialize module: failed to call module "test" to get functions: call constructor: process "go build -o /runtime ." did not complete successfully: exit code: 1
                                
                                Stderr:
                                main.go:3:8: package dagger/test/internal/dagger is not in std (/usr/local/go/src/dagger/test/internal/dagger)
                                1   : 2fedb8a31a0e3ca7: initialize ERROR [3.2s]
                                1   : 2fedb8a31a0e3ca7: ! input: module.withSource.initialize resolve: failed to initialize module: failed to call module "test" to get functions: call constructor: process "go build -o /runtime ." did not complete successfully: exit code: 1
                                
                                Stderr:
                                main.go:3:8: package dagger/test/internal/dagger is not in std (/usr/local/go/src/dagger/test/internal/dagger)
                                
                                Error: input: module.withSource.initialize resolve: failed to initialize module: failed to call module "test" to get functions: call constructor: process "go build -o /runtime ." did not complete successfully: exit code: 1
                                
                                Stderr:
                                main.go:3:8: package dagger/test/internal/dagger is not in std (/usr/local/go/src/dagger/test/internal/dagger)
                Test:           TestLegacy/TestExecWithEntrypoint
#

package dagger/test/internal/dagger is not in std huh, why should it be in std?

timber spade
timber spade
round geode
#

mmmmmm

timber spade
#

idk why for 100% sure, but might be that if an import path doesn't start with a URL-looking thing and the import path doesn't exist in the codebase, then it assumes it must be from the stdilb

round geode
#

the problem is. the module is called test, the path is test

timber spade
#

This does sound an awful lot like the python flake and the one you mentioned above

#

like different heads of the same monster

#

(could be wrong, just a suspicion)

#

Like if the Directory is just missing files or dirs that are supposed to be there, that feels like it could explain all of them

round geode
#

mmm i wonder how that could possibly be happening

timber spade
#

I don't know, that's why I gotta repro locally, but "thankfully" this one appears to be easier to trigger locally than the python one maybe?

round geode
#

yeah, i can get weird things to happen just running all of TestLegacy together over and over

timber spade
#

Trying now (really really hoping this isn't another edge merging issue, but that is coming to mind as a possibility in that it could swap one directory for another if incorrectly merged)

timber spade
round geode
#

i was doing dagger --progress=plain -m dev call --source=.:default test custom --pkg=./core/integration --run="TestLegacy"

#

it's potentially possible there's something weird about my setup

timber spade
#

I started with ./hack/dev + go test and ran it 50 times in a row successfully. Trying dagger call now

#

Oooo

Stderr:
12  : [57.3s] |                                 main.go:3:8: package dagger/test/internal/dagger is not in std (/usr/local/go/src/dagger/test/internal/dagger)
round geode
#

🎉 i'm not crazy!

timber spade
#

Just in terms of time, it's super late for you Justin and you mentioned you were still working on splitting out changes from https://github.com/dagger/dagger/pull/7858, right? At this point that latest I would feel comfortable starting the release is in an hour (3 my time) so idk if it's realistic to get everything sorted by then? Should we just plan to start it in EU tomorrow and hand over to US?

round geode
#

sorry, i was distracted writing tests

timber spade
#

Awesome, no worries, looking now. There's the docs PR too right? But that's it?

round geode
#

i suddenly became convinced that this was very tricky logic and i didn't trust myself to have done it right

#

the one other thing i would have tried to do, but don't really have time now, was to prep release notes - i was gonna try and tidy up the summaries, and for the breaking changes at least, add information on how to migrate

timber spade
# round geode the *one* other thing i would have tried to do, but don't really have time now, ...

I am happy to tidy up. For instructions on how to migrate, some of them are pretty obvious and I can add high level descriptions there. But ideally we'd have something fleshed out with examples in each language, etc. Which is just going to take a while.

So either:

  1. I can start the release after the above PR is merged (shouldn't take past 3 my time unless I find anything major), format the release notes nicely to the extent feasible and then we follow up in the next few days (modulo weekend) with more instructions
  2. We delay release, write up full instructions, then do the release
#

I'd vote 1, but if others have opinions let me know

round geode
#

i'd vote 1 too - i think getting it out would be great

timber spade
round geode
#

yeah, looking at those now 🙂

#

then i'm gonna go 😄

#

i'm kinda worried the behavior is legitimately wrong - there's a thing where if there is no engineVersion, it should serve v0.9.9

#

since we used to not generate the engineVersion

timber spade
gloomy oasis
round geode
craggy osprey
round geode
#

i'm off - pizza beckons goosepizza

#

i'll be around tommorrow morning to pick up anything, and help people who are trying to do migrations 😄

high ember
round geode
#

i cleaned up the milestone, i think everything in it should be good to merge now (assuming tests etc)

timber spade
# round geode right, i pushed some final fixes to this, hopefully that does it

From a read it LGTM, provided tests pass I'll approve and merge. If there's anything more though I will fix it but would then be getting a bit tenuous in terms of doing the release fully. Realistically I'd just need to take an hour-ish break at some point, so provided there's not somehow a time-sensitive fire that should be okay

sullen zephyr
#

Guys...

#

(writing docs over here)

timber spade
#

The tests still failed legitimately in that PR, looking

gloomy oasis
#

merged the JSON fix

round geode
#

Looks good to have taken that PR - one small note before I definitely well and truly go to bed 🛏️, before tagging and releasing, it would be good to sanity check a cli and engine with a semver --version to make sure it doesn't all just fall apart

timber spade
hard shoal
#

Hey, I'm back and around for maybe 1h if anything comes up.

timber spade
# hard shoal Hey, I'm back and around for maybe 1h if anything comes up.

Was just gonna say I saw a legit failure in the python runtime module, but then saw you pushed, so presumably fixed 😄

We need the change to Void of course, but is the python runtime module PR 100% necessary for v0.12 or can it wait for 0.12.1? Just if it comes down to making a call there

hard shoal
#

What's the failure?

#

bookworm?

timber spade
# 
marshal: json: error calling MarshalJSON for type *dagger.GeneratedCode: input: container.from resolve: failed to resolve image docker.io/library/python:3.10.10-slim-bookworm: failed to resolve source metadata for docker.io/library/python:3.10.10-slim-bookworm: docker.io/library/python:3.10.10-slim-bookworm: not found
#

Yeah

hard shoal
#

Yeah, that's reverted. Been having flakes until now. Hoping for 🥦

timber spade
hard shoal
#

call function "TestPublish": context canceled grrr

#

CI has been very frustrating lately. When things fail so much you start learning to ignore them which isn't good.

#

For the ProjectLayout flake we can add a few t.Log to get the state of the container in a few places. If expected files are in place and with the right contents for example.

timber spade
#

We never got to 0 flakes, but like 2 weeks ago they were relatively rare IME, but in the last week more popped up. It's mostly been the weird typescript stuff for me, but there's definitely others

timber spade
hard shoal
timber spade
hard shoal
#

Yeah, I noticed that!

#

I wonder after the OTel instrumentation in the test suite @gloomy oasis, where connect() should go, because of the pararelization? There's some tests with only a connect at "top-level" and then multiple sub-tests with the calls. Previously you'd need every subtest to have their connect if you wanted to use t.Parallel().

gloomy oasis
# hard shoal I wonder after the OTel instrumentation in the test suite <@108011715077091328>,...

Yeah, it's technically a regression to the time before Justin refactored t.Parallel()/connect() usage. I've thought about moving it into middleware or something so it only happens just before each test runs. It's not super clear whether holding those sessions open is a genuine issue or theoretical though (even at the time of the original refactor afaik)

tangent: right now having a top-level connect() also breaks the span hierarchy since everything ends up beneath the dagger session CLI instead of beneath each sub-test, which I have a local fix for, but need to isolate to the minimal change.

hard shoal
#

So it's best not to share connect() then?

gloomy oasis
#

yeah. afaik the downside is minor, but it's technically better to do it in each sub-test

hard shoal
timber spade
#

Manually setup v0.12.0 engine+CLI locally, can call unmodified CI module functions fine, run module tests configured to use those fine, etc.

Then did dagger develop to upgrade CI module to v0.12.0, which worked, but then when I try to call engine lint this is the only error I see by default:

sipsma@dagger_dev:~/repo/github.com/sipsma/dagger$ _EXPERIMENTAL_DAGGER_RUNNER_HOST=docker-container://dagger12 ~/dagger call -m dev --source=.:default engine lint
Setup tracing at https://dagger.cloud/traces/setup. To hide: export SHUTUP=1

✔ connect 0.2s
✘ initialize 1.6s
! input: module.withSource.initialize resolve: failed to initialize module: failed to call module "dagger-dev" to get functions: call constructor: process "go build -o /runtime ." did not complete successfully: exit code: 1
  ✔ resolving module ref 0.1s
  ✘ installing module 1.4s
  ! input: module.withSource.initialize resolve: failed to initialize module: failed to call module "dagger-dev" to get functions: call constructor: process "go build -o /runtime ." did not complete successfully: exit code: 1
    ✔ ModuleSource.resolveFromCaller: ModuleSource! 0.2s
    ✔ ModuleSource.asModule: Module! 1.0s
    ✘ Module.initialize: Module! 0.3s
    ! failed to initialize module: failed to call module "dagger-dev" to get functions: call constructor: process "go build -o /runtime ." did not complete successfully: exit code: 1

If I set --progress=plain then I get the actual incompat problems:

Stderr:
# github.com/dagger/dagger/dev
./helm.go:20:10: undefined: Directory
./helm.go:33:25: undefined: Container
./helm.go:46:5: undefined: File
./helm.go:71:50: undefined: ContainerWithNewFileOpts
./cli.go:162:36: cannot use dagger.ContainerWithNewFileOpts{…} (value of type dagger.ContainerWithNewFileOpts) as string value in argument to ctr.WithExec([]string{…}).WithDirectory("/nix", dag.Directory()).WithNewFile
./cli.go:163:4: unknown field Contents in struct literal of type dagger.ContainerWithNewFileOpts
./main.go:110:29: undefined: GoLintOpts
./sdk_elixir.go:114:34: cannot use dagger.ContainerWithNewFileOpts{…} (value of type dagger.ContainerWithNewFileOpts) as string value in argument to ctr.WithNewFile
./sdk_elixir.go:115:4: unknown field Contents in struct literal of type dagger.ContainerWithNewFileOpts
./test.go:229:65: unknown field Contents in struct literal of type dagger.ContainerWithNewFileOpts
./test.go:229:65: too many errors
#

Those are the known incompatibilities, but the fact that I can't see anything unless I do --progress=plain seems wrong. Was this a known issue? cc @gloomy oasis

hard shoal
#

Yeah, I've hit that a lot. I mentioned it on Discord.

#

Didn't have time to file an issue.

gloomy oasis
#

hmm maybe it's not revealing internal spans when it should? I disabled the top-level error log reporting since it's in principle redundant with the progress report and we have that longstanding issue to dedupe all that. But seems like there's a gap here

hard shoal
gloomy oasis
#

specifically I think calls beneath Module.initialize are marked internal: true because they're dagql.Select "self" calls

timber spade
#

My main concern is just that it's gonna make upgrading even more painful. I can just put it in bold in the release notes. But if there's any super low risk + quick ways to fix it, seems significant enough to do

gloomy oasis
#

but we should probably have those be revealed if they're in error state

#

let me see if there's a quick fix - do you have a repro available?

hard shoal
#

Early this morning you could have a missing required flag error, and it didn't show with TUI.

#

Like dagger-dev call -m dev

#

Or an error with the value to -m.

gloomy oasis
hard shoal
#

Oh that came after. I'm rebuilding dev and trying it out again.

gloomy oasis
timber spade
# gloomy oasis let me see if there's a quick fix - do you have a repro available? 
dagger call -m dev --source=.:default --version v0.12.0 engine container --platform linux/arm64 export --path=/home/sipsma/engine.tar
dagger call -m dev --source=.:default --version v0.12.0 cli file --platform linux/arm64 export --path=/home/sipsma/dagger
docker load -i /home/sipsma/engine.tar
docker tag a1ed7546b53f1170739034d353a649740e19e8b19535fc0f091bfe9cd62e0e69 dagger12
docker run -d -v dagger12:/var/lib/dagger --name dagger12 --privileged dagger12
_EXPERIMENTAL_DAGGER_RUNNER_HOST=docker-container://dagger12 ~/dagger develop -m dev
_EXPERIMENTAL_DAGGER_RUNNER_HOST=docker-container://dagger12 ~/dagger call -m dev --source=.:default engine lint

(highly manual to get a semver tagged local engine + cli right now)

hard shoal
#

Also was having issues using the python runtime module directly. Some errors flashed in TUI and disapeared in the end, but showed in progress=plain.

gloomy oasis
timber spade
timber spade
#

Alice just reminded me that besides picking out houses to tour by tonight we also have a zoom meeting w/ another realtor at 6..... so we'll see what I can do. I just don't wanna push a tag, disappear for an hour and come back to everything broken at 7pm lol.

I'll at minimum do the release notes since that's before any tag pushes. I'll decide then whether to keep going or wait for EU to pick up

sullen zephyr
#

zero pressure on my end to rush this

timber spade
#

Working on the notes now, it would be very nice to link to the blog post where there's more details on how to fix the various incompatibilities, but obviously a bit of a catch-22 😄

So I think we can just edit the release notes after it's out to link to that.

Another thing I just realized, it would be especially bad for us to have a long time gap between the Engine+CLI release and daggerverse being updated in this case. Due to the fact that module publishes using v0.12.0 exclusive APIs will fail until daggerverse is upgraded (right? I'm not misthinking about this?)

So from that perspective I think we do have to wait until it's working hours for infra+cloud people to push the engine tag. Obviously we can't avoid any gap between engine release + daggerverse updates right now, but should at least keep that minimal.

  • Also of course worth thinking about in the long term how to get rid of this problem, but for another time
#

Copilot's suggested autocompletion in the release notes made me laugh here 😄

Screenshot_2024-07-11_at_5.19.13_PM.png
timber spade
#

(Gonna be afk for a bit, on return I'll get as much ready as possible and leave a hand-off comment summarizing all the above)

timber spade
#

Back, gonna do some more manual testing of version/compat stuff for now

timber spade
#

Tested compat stuff with my fake v0.12.0 engine+CLI and went through updating all our ./dev + submodules to v0.12.0 gradually. Went super smoothly 🎉 Could go one at a time and dependency modules were still working when using v0.11.9

To save time for others later when we need to actually do this, diff is in links below

#

To summarize the current state of things for Justin tomorrow:

I'm gonna spend a little time poking at that flake in TestLegacy/PythonLayout around missing files now because it is proving to be highly annoying in CI. If I find anything before bed will update. EDIT: did make some progress actually, opened issue here with what I found so far: https://github.com/dagger/dagger/issues/7900

round geode
#

awesome, thanks for the writeup - i had a small realization that some of the tests i threw together last night will actually fail on tags - which isn't terrible, but still worth fixing

#

so i'll put that together, and take over the release notes

#

idea is to tag shortly after my lunch here, hopefully this maximizes number of folks online

round geode
sullen zephyr
#

Hi @round geode just a reminder that I'm available today to help

#

focusing on the docs and marketing side of things at the moment

round geode
#

while waiting for tests to pass, will go grab lunch

sullen zephyr
#

jumping on a call right now, will get to this queue right after

round geode
#

@hard shoal you around to look through the above?

hard shoal
#

Yeah, in a minute.

#

@round geode, are the backlashes needed in the changelog?

round geode
#

the other option apparently is to leave two spaces at the end of a line

hard shoal
#

I see, so it creates a <br>.

round geode
round geode
hard shoal
#

Yeah, good call.

#

There's a few changes I want to suggest, so it'll take a minute.

round geode
#

yup, ofc 🙂

noble basin
#

Anything that I can help with @round geode @hard shoal ?

round geode
#

^ some of the linked above prs

round geode
round geode
sullen zephyr
#

back

round geode
#

woop, well all prs are in now 😄

sullen zephyr
#

That's a question for @lean nest or @kind flame

noble basin
sullen zephyr
#

@silent @lean nest @kind flame Can we remove "release" from that graphic

#

It's just Dagger 0.12

round geode
#

going to tag in 5 mins

#

fyi @noble basin @hard shoal @sullen zephyr - speak now or hold your peace 😄

#

i should have enough time to run the whole release - but any follow-ups etc, will need to be handle by the us team

sullen zephyr
#

all good here 🙂

round geode
sullen zephyr
#

Oh god KSP

lean nest
round geode
#

since the pr failed to auto-generate

sullen zephyr
#

@round geode done

round geode
#

just updating it with some more things

sullen zephyr
#

Yeah @round geode @hard shoal @timber spade @gloomy oasis just to prepare you for what's coming: people are definitely going to get tripped up by the combo of 1) compat mode kicking in without their knowledge 2) new Terminal() mysteriously not working

#

I just hit it again

#

Not impact on the release itself, just preparing you for the post-release wave of support

#

It's compounded by the fact that the error isn't a clean Terminal: never heard of that. It's actually Terminal exists but it doesn't have this method you're trying to use (because it's a breaking change and the old Terminal type doesn't support chaining)

#

So the real problem is further obsfucated

round geode
sullen zephyr
#

@round geode any chance you could merge my docs PR too? 🙂

#

(that way we sneak it in the same manual docs deploy)

round geode
#

just scanning through now

#

one little typo, otherwise lgtm

sullen zephyr
#

thanks, merged!

#

@round geode do you have a prod docs deploy in your launch checklist?

round geode
#

yup, i'm just running down the RELEASING doc

#

it's quite near the end

sullen zephyr
#

OK then, one less thing on my own list

round geode
#

really don't love the fancy dance around prs

sullen zephyr
#

@round geode before I write that comment in the PR: is it still the case that removing auto-cli-download from the SDKs would solve a lot of pain on the engineering and release side?

#

(Impact on the user would be: your custom client now has a new runtime dependency: the dagger CLI. If it's not there, you get dagger: command not found)

#

Asking since it seems related to 7705

round geode
#

yeah, it definitely would - tbf, now is exactly the right time to bikeshed removing that in v0.13 😄

sullen zephyr
#

then let's do it

round geode
#

sadly, we still need that little dance

#

or maybe not

#

when my head is clear again on monday i shall have an answer 😄

sullen zephyr
#

Anything that simplifies the infernal version matrix is a win

round geode
#

agreed. i've now just become convinced that versioning is actually one of the hard problems

#

oop

#

Error: input: moduleSource.withContextDirectory.asModule resolve: failed to create module: select: failed to update codegen and runtime: failed to generate code: failed to get modified source directory for go module sdk codegen: select: error committing 1iglevyg6wrglgvs8vf22ehgg: database not open

sullen zephyr
round geode
#

thankfully. unrelated to the recent release i think, lint job is still using 0.11.9

#

hm that's weird though, because it's not like it was one worker that saw the issue

#

could potentiallllly be some weird disk flake

#

it's from buildkit, but i've never seen this before ever

#

re-running and hoping to never see it again

noble basin
sullen zephyr
round geode
#

finished the sdks

#

hit a small hiccup with helm, need to do a pr, and tag that

lean nest
round geode
#

going to do the docs update

sullen zephyr
#

Let me know when it's safe to post about it!

round geode
#

@dapper socket @ocean egret @noble basin @compact lava the playground, daggerverse, cloud, etc should all be good to upgrade to v0.12.0

#

@noble basin @ocean egret install.ps1 and install.sh need updating on cloudfront

lean nest
hard shoal
round geode
#

docs should be live @sullen zephyr!

#

happy to share more widely now, the only missing piece is helm

#

and github actions

lean nest
lean nest
#

@hard shoal can you refresh, and see if that fixes the issues you saw?

hard shoal
noble basin
round geode
#

I'm heading off for the weekend in about half an hour FYI y'all

noble basin
#

dagger-for-github is done

#

doing install now

#

helm PR approved & merged

#

anything else in the queue?

sullen zephyr
#

@round geode @noble basin if I tweet about the release right now, is it bad?

noble basin
#

OK, install.sh is now up-to-date. Want to confirm @round geode ?

round geode
#

yup!

sullen zephyr
#

so I can tweet? 🙂

round geode
sullen zephyr
#

well I just lost my whole twitter thread

noble basin
sullen zephyr
#

literally 30mn of work

#

and Coco just arrived

round geode
#

also have the first bug of v0.12.0 🎉

noble basin
#

OK, v0.12.0 looks correct at the surface.

round geode
#

@noble basin are our ci workers healthy? noticing some v0.11.9 workers having more issues than usual

round geode
noble basin
#

Checking

#

Jumping in team audio to make this quicker

round geode
#

could there maybe something where doing the release spins up a ton more workers? and this then overcrowds nodes

hard shoal
#

github.com/dagger/dagger/dev/wolfi@v0.12.0?

#

Problem with a std lib in our main repo (to be used externally) is that it takes a long time to load. Still going

Screenshot_2024-07-12_at_16.32.26.png
#

Took 2m39s

#

We need to check the modules used in our docs.

gloomy oasis
#

on that note, worth bumping its /apko dependency, I just bumped it + Bass to Dagger v0.12 which 1) removes that noisy checkVersionCompatibility, and 2) fixes telemetry in the Bass SDK so you can actually see what it's doing

#

I can put up a PR for that

sullen zephyr
#

@hard shoal I would avoid referencing modules in dagger/dagger for now.

#

That will one day become our stdlib, but it's too soon, we need the flexibility to move things around

hard shoal
#

Just asking which wolfi is updated to use externally.

#

Seems your repo has it fixed, just needs to publish (needs a tag).

#

Oh, but might as well also update the apko dependency like Alex said.

gloomy oasis
hard shoal
round geode
#

fingers crossed it's a particularly weird infra flake outside of our control, but given i've never seen this, i really don't have any good ideas (cc @timber spade @gloomy oasis who may have seen this weird buildkit-y error before)

hard shoal
sullen zephyr
#

Didn't realize that was in there

hard shoal
sullen zephyr
#

Yeah hello is fine

#

I added github.com/shykes/hello but didn't touch the daggerverse one

hard shoal
#

But needs a version bump in the doc.

# 
Stderr:
# hello
./main.go:38:19: undefined: Container
sullen zephyr
hard shoal
sullen zephyr
#

@hard shoal I added you as collaborator on both repos

hard shoal
#

Modules from our team are likely to have dev versions in them so it's breaking in the docs rn.

sullen zephyr
#

in case you need to rush a change

round geode
#

i'm off now salute i'd say ping me if it's urgent, but i'll probably be napping 🙂 (staying up late last night to finish all the bits off is taking it's toll now)

noble basin
# round geode <@796825768600141844> are our ci workers healthy? noticing some v0.11.9 workers ...

I did a some digging and concluded that this is a side-effect of heavy contention. When there are many CI jobs using the same Dagger Engine, our pipelines will flake. I just watched the same job pass when there is a single Engine & CLI test running on a single Engine. It passed the first time in 10 mins.

When we were seeing the failures, we had 8 CI jobs running at the same time and then the load looked like this:

image.png
timber spade
# round geode fingers crossed it's a particularly weird infra flake outside of our control, bu...

Yeah never saw that before either. It's from boltdb: https://github.com/sipsma/buildkit/blob/d2c730c0f9ab50dd4178f84dd8c75392d4062254/vendor/go.etcd.io/bbolt/errors.go#L9-L9

With the buildkit part of the error here: https://github.com/sipsma/buildkit/blob/d2c730c0f9ab50dd4178f84dd8c75392d4062254/solver/llbsolver/ops/exec.go#L396-L396

The thing is, if you are that far along in the buildkit code, I think the boltdb must have been open and at least read successfully by that point. Which, if correct, would imply that it's suddenly closing.

Running out of disk comes to mind as a possibility, but just a guess. Not sure what boltdb does in that case off the top of my head

#

Also possible the engine got a SIGTERM, started shutting down (including closing DBs), but this particular operation hit this error before anything else and managed to make it out in time to the client?

hard shoal
#

👉 https://github.com/dagger/dagger/pull/7909

I've updated @sullen zephyr's modules, but there's a lot of references to @gilded grail github.com/kpenfound/dagger-modules/golang and that's failing due to Go version.

docs/current_docs/integrations/snippets/Jenkinsfile:1
docs/current_docs/integrations/snippets/actions.yml:1
docs/current_docs/integrations/snippets/argo-workflow.yaml:1
docs/current_docs/integrations/snippets/azure-pipelines.yml:1
docs/current_docs/integrations/snippets/buildspec.yml:1
docs/current_docs/integrations/snippets/circle.yml:1
docs/current_docs/integrations/snippets/gitlab-docker.yml:1
docs/current_docs/integrations/snippets/gitlab-kubernetes.yml:1
docs/current_docs/integrations/snippets/tekton-dagger-task.yaml:1
docs/current_docs/manuals/user/artifacts/consumption/export.mdx:3
docs/current_docs/manuals/user/artifacts/production/directories.mdx:1
docs/current_docs/manuals/user/artifacts/production/inspect.mdx:1
docs/current_docs/manuals/user/functions/arguments.mdx:1
docs/current_docs/manuals/user/functions/chaining.mdx:1
docs/current_docs/manuals/user/host/host-fs.mdx:1
docs/current_docs/manuals/user/remotes/remote-repositories.mdx:1
docs/current_docs/manuals/user/visualization/cloud-get-started.mdx:1

I have to go away for the weekend now! 👋

lean nest
#

@craggy osprey not sure if you can help for the Kyle stuff

craggy osprey
#

@noble basin thanks for merging the dagger-for-github PR to upgrade to v0.12, I'm going to increment the action version from v5 to v6 to reflect

#

currently it has v5.12.0 any objection to moving that to v6.0.0 @round geode

#

since there are breaking changes
I'll hold off in case we're using the action and depending on the version

craggy osprey
# 
dagger-fork ➤ grep -r dagger-for-github@v5                                                 git:main
./docs/current_docs/integrations/snippets/google-cloud-run/main.yml:        uses: dagger/dagger-for-github@v5
./docs/current_docs/integrations/snippets/actions-ghcr.yml:        uses: dagger/dagger-for-github@v5
./docs/current_docs/integrations/snippets/actions.yml:        uses: dagger/dagger-for-github@v5
#

just docs, it looks like

#

@ocean egret looks like we're not depending on dagger-for-github action in our CI, correct?

#

So I'll move forward with bumping our action to v6

sullen zephyr
#

Amazing work people! Thank you so much

#

and a special thank you for @round geode for running this release like a boss

dapper socket
#

bumping Daggerverse to v0.12.0 so we don't overlap cc @ocean egret @craggy osprey

dapper socket
dapper socket
lucid birch
#

@dapper socket yeah I raised the same issue yesterday, those aliases are gone

dapper socket
#

not sure what needs to be changed in Daggerverse to still make it initialize older modules? Haven't followed in-depth the compat mode thing.

lucid birch
#

Ohh ok I thought you were referring to daggerverse's pipelines

#

this is actually daggerverse failing the mod builds

dapper socket
#

gotta run now, will check back after dinner 🙏

craggy osprey
#

I’ll jump on after a bit too. Guessing we’ll need to update all the daggerverse modules internally, yep

round geode
#

engineVersion should not be a dev version

#

@sullen zephyr mentioned this earlier in the thread, but looks like this one got forgotten

#

If engineVersion is a dev version, the only possibly interpretation of this is to serve the latest API (there's not really anything clever we can do here, sadly until the semver everywhere work gets done and merged)

#

If it doesn't, I briefly looked through the daggerverse module loading flow, and it's honestly very unfamiliar 😢 I can take a look on Monday potentially, maybe there's some bug on dagger / daggerverse about the way module init works (sorry, on mobile so reading code is hard)

#

Note for the future - because of module compat, we should actually be able to bump daggerverse before a release, so we can avoid a scramble after

#

I guess we are in a pickle about what to do with dev versions now, since technically we have no idea whether they're v0.11 compatible or v0.12 compatible 🤔
I guess we should attempt to purge them as much as possible from the daggerverse, but not sure what we should do if we encounter them.

#

I kind of wonder if daggerverse should not allow users to publish modules with a non semver version at all going forward (unrelated, just musing)

round geode
dapper socket
#

Thx for replying so late your time Justin

round geode
#

Sorry it's all on mobile and unstructured

#

🙂

dapper socket
#

I was actually aware of this but totally forgot to check this module's dagger.json

#

For some reason I assumed it should work given that's the module that we use pretty much anywhere

#

I guess we should disallow indexing modules in Daggerverse without a semver engine version then

round geode
#

Or at least, in the case that it fails, modify the error message to include a warning about semver as well

hard shoal
#

I didnt update shykes/hello because what’s on our docs is shykes/daggerverse/hello. That one I updated and tagged. Didn’t check if we still auto publish or not but tested directly

dapper socket
craggy osprey
#

Working on a problematic part of Daggerverse where we're doing a dag, err := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))
and then using alpine/git doing WithExec([]string{"show", "--no-patch", "--format=%ct", commit}).Stdout(ctx) essentially a git show.

Stderr:
[dumb-init] show: No such file or directory

So we're hitting Skip entrypoint by default in WithExec()https://github.com/dagger/dagger/blob/main/.changes/v0.12.0.md?plain=1#L12-L14

craggy osprey
#

Confirmed. I have to change the code to one of these:
WithExec([]string{"show", "--no-patch", "--format=%ct", commit}, dagger.ContainerWithExecOpts{UseEntrypoint: true}) ✅
WithExec([]string{"git", "show", "--no-patch", "--format=%ct", commit}) ✅

Unfortunately this shows up in my IDE:
WithExec([]string{"show", "--no-patch", "--format=%ct", commit}, dagger.ContainerWithExecOpts{SkipEntrypoint: false}) ❌ which doesn't work

#

===
I used the top line and that seems to fix it, @dapper socket

#

Also changed this a bunch in the modules part of things. Not sure if necessary, but was following convention.

image.png
#

Though strangely the dagger.gen.go keeps reverting to

import (
  ...
  "main/internal/dagger"
  "main/internal/telemetry"
  ...
)
#

when I delete it and re-gen with dagger develop

#

even though

{
  "name": "daggerverse",
  "sdk": "go",
...
#

🤷‍♂️

#

Because in module I recently created called interactive I get this in dagger.gen.go:

import (
  ...
  "dagger/interactive/internal/dagger"
  "dagger/interactive/internal/telemetry"
  ...
)
#

oh! I get see the difference module github.com/dagger/dagger.io/daggerverse in our go.mod, versus module dagger/interactive in my new module's go.mod.

hard shoal
craggy osprey
# hard shoal Second form is preferred. Makes it explicit that it’s a git command so it’s easi...

I agree the second form is clearer if the entrypoint is simple like with this one:
https://github.com/alpine-docker/git/blob/master/Dockerfile#L10
ENTRYPOINT ["git"]

But I suspect with DBs for service containers and some other cases, the first form will get a lot of use, which will make that use case more verbose:
https://github.com/docker-library/postgres/blob/master/Dockerfile-alpine.template#L219
ENTRYPOINT ["docker-entrypoint.sh"]
https://github.com/docker-library/postgres/blob/master/docker-entrypoint.sh 356 line script with vital setup

#

Oh, maybe not, because we won't need WithExec() on the service container itself perhaps...

#

Still need some coffee. Early here. Trying it out.

hard shoal
# craggy osprey I agree the second form is clearer if the entrypoint is simple like with this on...

There’s things you can do programmatically. If you have a common start you can put it in a var. If you still need to set the option you can put that in a var and reuse to reduce verbosity:

opts := dagger.ContainerWithExecOpts{
    UseEntrypoint: true,
}

ctr.
    WithExec(…, opts).
    WithExec(…, opts)

See changes in Redis snippets in https://github.com/dagger/dagger/pull/7905/files

GitHub
docs: update docs with breaking changes from v0.12.0 (part 2) by he...

Continuing from:

#7884

TODO:

Re-test printing objects with properly tagged cli (had issues with dev due to daggerverse dependencies)

craggy osprey
#

Like this example I had

// Database service used for application tests
    database := dag.Container().From("postgres:latest").
        WithEnvVariable("POSTGRES_PASSWORD", "test").
        WithExec([]string{"postgres"}).  // <<<< this breaks
        WithExposedPort(5432)
        AsService()

// Run application tests
    out, err := client.Container().From("golang:1.22").
        WithServiceBinding("db", database).     // bind database with the name db
      ...
#

would need to update the WithExec either way...got it

var execOpts = dagger.ContainerWithExecOpts{
    UseEntrypoint: true,
}
...
  WithExec([]string{"postgres"}, execOpts).
...
#

or

...
  WithExec([]string{"postgres"}, dagger.ContainerWithExecOpts{
    UseEntrypoint: true})
...
craggy osprey
craggy osprey
round geode
#

🚀 v0.12.0 - 9th July 2024

round geode
#

just need to bump our ci workers to use the latest release 😄

#

uh oh

#

from that pr, noticed that some of the jobs seem to terminate, and then never actually exit

#

fyi @gloomy oasis @timber spade, fingers crossed it's not telemetry draining again

gloomy oasis
round geode
#

Mm but looking at the timestamps, the job finished way before the cancellation

#

It could somehow still be that ofc

gloomy oasis
#

does the cancel come first, or does it happen when GHA realizes the runner is missing?

ocean egret
#

This is still in the works and is a part of the gen2 runners, but we have on demand runners in the new ci cluster: dagger-gen2-v<VERSION>-<size>-od. It is not considered production yet but soon

#

Feel free to use them in PRs if it helps you troubleshoot. We are working on them so expect some instability

round geode
#

still worth keeping an eye out for, if there is an issue we will definitely see it once we deploy more widely

craggy osprey
#

Including dagger -m github.com/dagger/dagger/linters/markdown@88d89e8d15ab6ad9ca4043a920d3cd735a6405fd call rules contents which I'm guessing doesn't work due to the change in CI structure. But looking now.

Can see it places like: https://docs.dagger.io/manuals/user/chaining/

This vestigial tail remains: https://github.com/dagger/dagger/tree/main/linters

As someone who does not know this module inside and out, I'm not sure how to get to a markdown linter. Would be cool to be able to search within the module or expand all the possible "call trees". I'll go look at code now.

dagger -m github.com/dagger/dagger/dev call --help
Full trace at https://dagger.cloud/dagger/traces/b88703c275a0b51c0d1b793f6d6233bc

✔ connect 1.3s
✔ initialize 3.7s
✔ prepare 0.0s

Call a module function

USAGE
  dagger call [options] [arguments] <function>

FUNCTIONS
  check         Check that everything works. Use this as CI entrypoint.
  cli           Develop the Dagger CLI
  dev           Creates a dev container that has a running CLI connected to a dagger engine
  docs          Develop the Dagger documentation
  engine        Develop the Dagger engine container
  go            Dagger's Go toolchain
  helm          Develop the Dagger helm chart
  scripts       Run Dagger scripts
  sdk           Develop Dagger SDKs
  test          Run all tests
  version

ARGUMENTS
      --source Directory    [required]
      --docker-cfg Secret
      --version string
craggy osprey
#

Except now, the CLI examples really want to be back...Have an issue for that, will prioritize.

image.png
#

So going to try that one.

image.png
#

So guess this is pretty similar area: dagger call -m github.com/dagger/dagger/dev --source https://github.com/dagger/dagger\#main docs lint
but we don't have a function to show the rules...so I'll likely put in a different linter example or anything that returns File contents, which was the example's intent.

craggy osprey
round geode
#

the call fails, but the job hangs on for another 8 minutes

image.png
round geode
#

hm, specifically there seems to be something that this occurs if an error occurs, a weird hang can appear

#

what's strange is how this failure doesn't actually seem to propagate anywhere

round geode
#

what in the actual world is happening

#

we never see an initialize DONE

#

in all the failure cases

#

but how? analyzing module does complete, those defers are right next to each other

#

that would definitely explain the hang - if the initialize never hangs up somehow, then we would hang until it's completed

round geode
#

there's a withexec that seems to fail deep in the tree, that isn't propagated upwards

gloomy oasis
round geode
#

ohhhhh i forgot that that's a thing 👀

#

i guess it's a visualization level thing? can it be disabled in cloud?

#

also, this is totally the wrong place to discuss this one, so gonna spin up a new 🧵

kind flame
#

I noticed our docs need to be updated to mention multiple git servers...I tried to just edit it/submit a PR but I'm getting an error and not sure how to resolve. Any help would be much appreciated. @void pendant @dapper socket @compact lava

image.png
compact lava
kind flame
#

FYI if folks were curious, stupid marketing guy didn't realize he needed to fork the repo to submit a PR...🤦‍♂️ Feel free to make fun of me now.

kind flame
#

@lucid birch @void pendant @lean nest Jeremy and I just noticed we don't seem to have an docs for interactive debugging... seaching for terminal brings up the old way of doing things, there's a debugging section in docs with nothing on the new feature. We're curious if there's a plan to add these. I'm looking to add Interactive Debugging to our website as I think it's a really compelling feature but I think it's important that I link to docs so we don't fumble the conversion etc. I can certainly hold putting it up if folks feel its better.

lean nest