#✨ v0.12.5 - 15th August 2024

1 messages · Page 1 of 1 (latest)

weak laurel
#

I think aiming for v0.12.5 on Tuesday is good, but we could also stretch it to Wednesday/Thursday if we like:

#

cc <@&946480760016207902>

#

@timber gyro ^ hopefully this should get everyone? or is this missing some folks (I can't see who's actually in this role)

mint panther
#

Would love to get:

  • dagger core cc @round basin
  • context dir cc @rocky pecan
weak laurel
#

I've added dagger core to the milestone - I think it's good-as-is, but maybe we want to mark it as experimental? no strong opinions anymore on any of it, i just want it in 😄

round basin
#

dagger core will be merged today. As for the enums in constructor I asked Tom to work on it. I’ll leave on PTO for 2 weeks after today.

round basin
mint panther
weak laurel
round basin
#

I’m ooo atm, so on my phone, will be back in about 1h or so.

wispy bear
round basin
# weak laurel I've added `dagger core` to the milestone - I think it's good-as-is, but *maybe*...

Since it was a quick change, I leave you guys with a PR to give a little bit more time to decide on the DX for core functions:

GitHub
[DISCUSSION]: move `dagger core` to `dagger call -c` by helderco · ...

This changes the way you call core functions in the CLI (currently dagger core <function>):

dagger call -c <function> → Core function
dagger call -m <module&...

sudden tiger
#

Another great release brewing 🚀daggerfire

next bear
timber gyro
timber gyro
sudden tiger
mint panther
#

for 0.13 you mean @timber gyro ?

timber gyro
mint panther
#

I think it's a great idea to start writing the draft now though.

timber gyro
mint panther
#

then anytime we feel like there's enough to talk about, boom we promote to 0.13 and publish the blog post

weak laurel
timber gyro
weak laurel
#

@rocky pecan how's context dir coming?

mint panther
#

Helder made it experimental so that it wouldn't hold up the release

weak laurel
#

ahh, so it shouldn't be in the milestone then?

mint panther
#

Not for 0.12.5

weak laurel
#

tentatively put it in for 0.12.6

rocky pecan
weak laurel
#

i think we should centralize the conversation around that in that issue, the package manager pr is unrelated to that general problem

#

it's good feedback though

mint panther
#

good reminder of why I opened that issue in the first place. And of the urgency of fixing it

mint panther
#

@weak laurel @rocky pecan I'm trying to understand that comment. He says "I followed the quickstart on my repo, and got lots of issues". But the quickstart says to run dagger init --source=./dagger. If he followed that, then his issue is NOT the same as 8123, ie: his repo was not littered. If that's true, then what issues did he encounter, and why is he upset?

weak laurel
#

I don't think it's the littering so it's not exactly the same as 8123 - it's that the project structure is unclear, which I think doing the solution proposed in 8123 helps solve

mint panther
#

But he mentions errors that led him to the TS pull request. Do we know what these errors are?

#

His problem seems to be bigger than confusion

#

"Everything works fine but I'm confused about the role of this dagger/ directory" seems like a weak reason to churn

mint panther
#

@rocky pecan I need help understanding the context of "drmikecrow"'s error

#

are you around today?

wispy bear
mint panther
#

Here's the core of what I need to understand:

Here's what I'm starting to believe:

It's almost impossible to install dagger within a folder with an existing package.json. It appears it must be 1 level above
If you try to install dagger in your folder (as I did), the SDK yarn install will update the yarn.lock, which breaks our CI later since we use yarn install --immutable)
Also, our top-level package.json has packageManager: "4.1.1" field, which causes the SDK install to fail before it even tries to execute

So it looks like even with --source=./dagger, things still break if you mix a TS module into a TS app?

#

Is this a variation of "don't reuse parent go.mod" in the Go SDK? eg. we don't create a new package.json by default, but should? Or is it worse than that?

#

If it's the latter, it calls into question to validity of what I propose in 8123

rocky pecan
rocky pecan
mint panther
#

So to make sure I understand: if I run dagger init --sdk=typescript --source=./dagger, and ./package.json already exists, then the Typescript SDK will:

A. Create ./dagger/package.json
B. NOT create ./dagger/package.json

(edit: sorry I meant package.json)

rocky pecan
#

I need to verify, but normally it shouldn't create the package.json

mint panther
#

OK, if it did create the package.json, would that solve the problem? In the case of Go, creating a nested go.mod is the fix

#

and trying to understand if the Typescript issue is a variation of the same problem ("don't reuse parent package.json" instead of "don't reuse parent go.mod"). You see what I mean?

rocky pecan
#

There shouldn't be any issue, even with a parent package.json, that's the thing :/

mint panther
#

But you said:

I think it's related to the same thing

But now you're saying it's NOT a variation of the same thing correct?

rocky pecan
#

What I'm saying is that, it's related to merging the parent or not, but in the case of TS, merging or not shouldn't affect the result

#

Oh wait, actually it should create an issue, because npm install ./sdk will set the deps to a local path, but if it's the parent package.json, the path will be wrong and this will fails

#

Like

.dagger
   package.json -> ./sdk
   sdk/
package.json -> ./sdk -> does not exist, it's .dagger/sdk so the path fails
#

So yes, that's my bad, it's totally related, however, I'm gonna verify what happens with a root package.json, give me 5 minutes

#

Ok, funny things, I did a test:

$ mkdir test-pkg && cd test-pkg
$ git init
$ touch package.json
dagger init --sdk=typescript --name=test --source.dagger
$ ls .dagger
package.json  sdk           src           tsconfig.json
$ cat package.json 
{
  "name": "project"
}
$ cat .dagger/package.json
{
  "dependencies": {
    "typescript": "^5.3.2",
    "@dagger.io/dagger": "./sdk"
  }
}
#

So even with a package.json at the context dir (parent level), there's no merge happening, so I would say it's a different issue in that case

weak laurel
#

👋 i'd like to see if we can delay v0.12.5 to wednesday - there's some work that redoes the otel pipeline https://github.com/dagger/dagger/pull/7996 by @wispy bear (and i'd kinda like the chance to dogfood it a bit)

#

dunno, maybe i'm just being overly worried from past otel draining issues 🙂

#

@marsh wyvern i can definitely help review the mod ref one tomorrow 🎉 (just about to head off for the day now)

#

just gimme a ping on it 🙂

marsh wyvern
wispy bear
prime axle
weak laurel
#

oh, it's not bass, that error looks fine my bad, somehow python then wraps it in a little box and messed up the line on the right side of the box

wispy bear
weak laurel
#

insane

image.png
wispy bear
#

it's probably including the escape codes in the width calculation for padding

wispy bear
#

@weak laurel fyi: merged the OTel PR 🎉

weak laurel
#

🎉

next bear
next bear
# rocky pecan Ok, funny things, I did a test: ``` $ mkdir test-pkg && cd test-pkg $ git init ...

@rocky pecan I was able to reproduce what the user was experiencing in v0.12.4

npm init -y 
corepack use pnpm@9.6.0
dagger init --sdk typescript --source dagger
dagger call container-echo --string-arg foo

^ this fails with:

Error: input: module.withSource.initialize resolve: failed to initialize module: failed to call module "l
olo" to get functions: call constructor: process "yarn install" did not complete successfully: exit code:
 1

Stderr:
error This project's package.json defines "packageManager": "yarn@pnpm@9.6.0". However the current global
 version of Yarn is 1.22.22.

this is my current directory layout:

.
├── dagger
│   ├── package.json
│   ├── sdk
│   ├── src
│   ├── tsconfig.json
│   └── yarn.lock
├── dagger.json
├── LICENSE
├── package.json
└── pnpm-lock.yaml

4 directories, 7 files

why is this failing? it doesn't seem right that the runtime container is trying to use pnpm when the module's package.json doesn't have a packageManager defined. Why is dagger call picking up the parent's package.json packageManager config?

I've tested with the latest version in main and the issue is still not resolved, something even more strange is happening. After doing dagger init , the root application package.jsongets overwritten with this value: "packageManager": "yarn@1.22.22+sha512.a6b2f7906b721bba3d67d4aff083df04dad64c399707841b7acf00f6b133b7ac24255f2652fa22ae3534329dc6180534e98d17432037ff6fd140556e2bb3137e"

weak laurel
weak laurel
rocky pecan
rocky pecan
rocky pecan
#

I’ll try to fix this today though, that’s quite strange

next bear
#

As it gets set by the corepack use

rocky pecan
#

That's strange that it's the parent package.json that is affected :

#

Normally the context dir is not even required to load the module, I only need the source dir

#

So I might have missdo something there

next bear
#

I have the impression that this whole packageManager thing was mostly a confusion

#

As you can see from my first example, v0.12.4 shouldn't fail as it shouldn't have to use the parent package.json at all

rocky pecan
#

Yeah I get it

#

I'm trying to solve that rn

#

Thanks for your repro!

weak laurel
#

since this looks like a regression for the typescript sdk, i think we should avoid releasing until we can fix it / revert the change

rocky pecan
#

I don't think it's a a regression, the issue has always been there but is more pronounced with the package manager feature, hope I fix this today

next bear
#

Will get to those as soon the nanny arrives 👶

rocky pecan
#

@weak laurel Could I get a review on that one please?

round basin
#

Should have addressed that feedback before merge.

weak laurel
#

there are many tests for how go module initialization works btw, in the future, it would be valuable to "copy" them, and check that typescript (and python tbf) follow the same style and conventions

round basin
rocky pecan
#

It's already removed

#

I only copy sdk/typescript/package.json so that's confusing, I think it's more related to context dir

rocky pecan
weak laurel
#

i also got a bit thrown by the github actions logs output no longer having any call logs 😦 (but ah well, i'll cope)

wispy bear
weak laurel
#

yeah, agreed, i guess i'd just kind of like to make the ui work, but it's not wrong to just disable it (and github log viewer just appears to be pain)

weak laurel
wispy bear
#

my guess is it's the 10 second otel draining timeout workaround cutting them off

#

not sure why there's so much lag, but it's a lot of telemetry volume

weak laurel
#

hm the Container.Sync seems to have finished according to the call logs

weak laurel
wispy bear
weak laurel
#

cool ❤️ yup, will dive into why typescript is trying to hang 😦

#

thanks for your help as always 🙂

wispy bear
#

i can take a look too, if you'd prefer i do that over PR reviews. i'm decently well trained on investigating hangs now 😛

#

i don't think this is otel draining at all, but maybe i'll see something familiar

weak laurel
#

yes please ❤️ all the help is appreciated 😄

#

release ain't happening today anyways 😄

#

fyi @rocky pecan on the above ^

rocky pecan
#

Strange 😮

#

And okay for the release

wispy bear
rocky pecan
wispy bear
#

@weak laurel thoughts on the hang so far:

  • the tests don't seem to have an obvious issue, BUT they do share a lot of common code between them and run in parallel, and they all use the same client, so likely opportunities for edge merging if that's relevant at all
  • there's no consistency as to which test that hangs - sometimes it's one, sometimes it's another, sometimes it's multiple
  • the engine doesn't seem to crash or anything (managed to open some logs and saw it just doing the engine-metrics logs at the end)
  • it seems like we've only seen this for test (and quite often) but never for testdev
prime axle
wispy bear
#

any info helps! 🙏

prime axle
#

I was offering to pair-up and start digging. Combining the two perspectives might be worth 1h or so.

wispy bear
#

another observation:

  • all hangs show that it's hanging in a call to Directory.export viadagger develop. (one hang is missing all telemetry for some reason but I'm assuming the same.)
#

they all look like this - unterminated export directory / to host /work

image.png
prime axle
#

looking at the host metrics in case there is anything interesting there

wispy bear
#

(once i find my headphones)

next bear
weak laurel
next bear
# weak laurel hm, can you re-run php generate? 
130|marcos:Projects/dagger (marcos/dev-4188-dont-reuse-parent-gomod-by-default) (⎈ |eks-prod)$ dagger  -m
 dev call --source .:default sdk php generate export --path .
Setup tracing at https://dagger.cloud/traces/setup. To hide: export GOAWAY=1

✔ connect 1.4s
✔ initialize 6.6s
✔ prepare 0.7s
✔ daggerDev(
    source: ✔ ModuleSource.resolveDirectoryFromCaller(path: ".", viewName: "default"): Directory! 0.0s
  ): DaggerDev! 0.2s
✔ DaggerDev.sdk: DaggerDevSdk! 0.2s
✔ DaggerDevSdk.php: DaggerDevPhpsdk! 0.0s
✔ DaggerDevPhpsdk.generate: Directory! 8.5s
✔ Directory.export(path: "."): String! 0.3s

/home/marcos/Projects/dagger
marcos:Projects/dagger (marcos/dev-4188-dont-reuse-parent-gomod-by-default) (⎈ |eks-prod)$ gs
On branch marcos/dev-4188-dont-reuse-parent-gomod-by-default
Your branch is up to date with 'origin/marcos/dev-4188-dont-reuse-parent-gomod-by-default'.

nothing to commit, working tree clean

🤷‍♂️

weak laurel
#

ah okay, taking a deeper look

next bear
#

no rush

weak laurel
#

yeah, i have to head out now, sorry 😦

#

yeah, not quite sure what's going on there 🤔

next bear
#

I can take a look, don't worry

weak laurel
wispy bear
weak laurel
#

so glad that we have at least one case of this happening before merging OTEL

#

but weird that there's no obvious cause 😢

wispy bear
#

got a stack dump, it is indeed locked up in the export path (second screenshot, corresponding to hangs in third screenshot), and there are some other funky looking things (first screenshot)

image.png image.png image.png
#

that massive amount of mutexes looks a little sus

#

will upload the full dump somewhere, it's 20MB lol

#

likely related goroutines stuck on chan send for 5 mins

image.png
#

here's where the other side of that chan send is stuck

image.png
weak laurel
#

Huh

wispy bear
#

also, line 214 in that last screenshot holds the same mutex that the very first screenshot are all blocking on

weak laurel
#

It's been around forever, we've just never seen it

wispy bear
#

possibly

#

i wonder if node_modules exists in this export and is what's triggering it?

#

since that's a whole universe of files

#

might explain why this test is uniquely running into it when we already have so many others that do dagger develop (including other TS tests)

#

also, the fact that this is happening 4x in parallel, all trying to go through the same client

#

short-term fix might be to just change the test to make a new client for each subtest, which might be a good idea anyway

#

or, yeah, fix the real issue somehow 😛

wispy bear
weak laurel
#

Ohhhh yeah, maybe we now export more files? That could do it

wispy bear
weak laurel
wispy bear
#

well we have an easy repro at least! my PR hit it 2/2 times after switching it back to running all tests. (curiously, it doesn't repro when running only that test)

wispy bear
# wispy bear trying this now

run succeeded, will give it another go to be sure. @weak laurel I can put up a PR for this fix to the tests to get the release unblocked, since this doesn't seem like a regression, just an accidental repro of a known issue 😛 - and then if you want to dig into the underlying issue, you could do it with a PR that reverts that change

weak laurel
#

Thanks thanks! 🙏🙏 Huge kudos for all of this

wispy bear
#

@rocky pecan [async] might also be worth taking a look to confirm that test is doing what you'd expect - just in case it's unintentionally picking up node_modules or some other unwieldy directory that doesn't actually need to be synced back

#

test passed again, gonna just turn my repro PR into a real PR

wispy bear
rocky pecan
rocky pecan
wispy bear
rocky pecan
# wispy bear no worries, the tl;dr is that test is somehow triggering a known Buildkit hang w...

Not as far as I know, I've been very cautious about node_modules exports, because it makes the SDK super slow.
If you find a way to export node_modules locally from the SDK to the host let me know.

However, I know that it's possible to do the opposite if no views is set (or contextual arg when it will be merge)
I'm still on it by the way, tracking down the path resolution...

It's a holiday day tomorrow but I can dedicate all my time on friday because there's nothing left on my list for the week

weak laurel
#

we merged it, and then it immediately continues to happen, on "should use pnpm if pnpm-lock.yaml is present" again 🤔

#

ah

#

i did a quick test

#

./.pnpm-store is exported by dagger develop

#

it's 124MB

#

@rocky pecan can we avoid exporting it? i feel like this is pnpm equivalent of node_modules

rocky pecan
#

You can add it in the TS runtime Codegen func, I can approve it when it's done

weak laurel
#

awesome, testing it now ❤️

rocky pecan
#

Lmk when I can review

weak laurel
#

✨ v0.12.5 - 15th August 2024

#

let's try and go for today

#

but then, i think we're good to go

#

@rocky pecan are you around today to help with pr approvals for the release process? should hopefully just be the release notes

rocky pecan
#

I can help with PR approval, it's a day off but I'm still on my computer haha

weak laurel
#

oh sorry! i missed it on the calendar, my bad

rocky pecan
weak laurel
#

right, starting to prep the release - everything in the milestone is now merged

mint panther
#

Thank you 🙏

weak laurel
weak laurel
weak laurel
#

engine + cli + sdks + chart + docs all done

#

@next bear @past hazel @prime axle @marsh wyvern
daggerverse and playground should be good to go

prime axle
weak laurel
weak laurel
prime axle
# weak laurel <@796825768600141844> anything i can do to help unblock here? looks like a user ...

While we could bump it, we don't know if v0.12.5 is going to break something for existing users. I think that we should hold off until we know that the new version is working as expected. I know that Matias is off today - he knows how to test Daggerverse (there is no integration yet). In his absence, who knows how we could test https://dgvs-add-option-to-preview-with-dagger-main.preview.daggerverse.dev/ ? cc @next bear @marsh wyvern @wispy bear

next bear
#

checking now

#

there's currently no official way to test backwards compatibility check

next bear
#

ok, ran a quick teste and we're good. PR incoming

prime axle
#

looking now

prime axle
charred gorge
charred gorge
# wispy bear and another on `main`, curiously misattributed to my PR's merge commit (cc <@445...

interesting, the metadata available in CH for that trace (5629f35bfe59bc3f4489dc9ca190e3e4) is

{
  "dagger.io/git.branch": "\"main\"",
  "dagger.io/vcs.workflow.name": "\"Engine \\u0026 CLI\"",
  "dagger.io/vcs.job.name": "\"test\"",
  "dagger.io/client.version": "\"v0.12.4\"",
  "dagger.io/vcs.repo.full_name": "\"dagger/dagger\"",
  "dagger.io/vcs.triggerer.login": "\"vito\"",
  "service.name": "\"dagger-cli\"",
  "dagger.io/git.author.name": "\"Alex Suraci\"",
  "dagger.io/git.remote": "\"github.com/dagger/dagger\"",
  "dagger.io/git.committer.email": "\"noreply@github.com\"",
  "service.version": "\"v0.12.4\"",
  "dagger.io/client.machine_id": "\"d16e27972d7862f6975a4a937a3536157aa5c146571830ed2b4bded668a52d64\"",
  "dagger.io/client.os": "\"linux\"",
  "dagger.io/ci.vendor": "\"GitHub\"",
  "dagger.io/git.ref": "\"718b5d7d7d6a4c41284b51d67e7f87940bee3fb1\"",
  "dagger.io/vcs.event.type": "\"push\"",
  "dagger.io/git.committer.name": "\"GitHub\"",
  "dagger.io/git.title": "\"redo OTel engine =\\u003e client pipeline (#7996)\"",
  "dagger.io/ci": "\"true\"",
  "dagger.io/client.arch": "\"amd64\"",
  "process.command_args": "[\"dagger\",\"-m\",\".\",\"call\",\"--source=.:default\",\"--docker-cfg=file:/home/runner/.docker/config.json\",\"test\",\"all\",\"--race=true\",\"--parallel=16\"]",
  "dagger.io/git.author.email": "\"suraci.alex@gmail.com\"",
  "dagger.io/vcs.repo.url": "\"https://github.com/dagger/dagger\""
}
wispy bear
# charred gorge interesting, the metadata available in CH for that trace (`5629f35bfe59bc3f4489d...

hmm, weird, if I go to the actual commit it links to two different traces (since there was a rerun):