#🚀 v0.15.1

1 messages · Page 1 of 1 (latest)

upper venture
#

👋 <@&946480760016207902> following up from #1309173342586277930, we're doing v0.15.1 very soon (today?)

whole fossil
#

This has now been confirmed as missing functionality in need of enhancement. Will be looking at a fix now.

upper venture
#

genius level move from me here 😛

rancid birch
#

@fierce heron do you want to give the running the release a try? I'm happy to help and be available for the whole process

upper venture
rancid birch
upper venture
#

there's also a 1password vault which has some credentials that are occasionally useful for emergencies

fierce heron
#

will take a look post-meeting

fierce heron
#

from the content, i don't see why not

rancid birch
fierce heron
#

i mean rebase the prep PR on main after you've merged

rancid birch
upper venture
#

yup, shouldn't be an issue with rebasing 😄

fierce heron
upper venture
#

@fierce heron i'm so glad it's close to a single script

#

that did not use to be true at all 😄

rancid birch
fierce heron
#

it's at the point where as i'm copy pasting the commands i am confused as to why they're not in a bash script with -ex pipefail, which is a very positive signal

rancid birch
fierce heron
#

yeah im following that now, saw jed post it

rancid birch
#

Let me know if you want me to be on standby in lounge or anything, happy to check stuff along the way if helpful

fierce heron
#

👍 discord calls are superborked for me today so it'll be fun if that's needed

upper venture
#

yeah feel free to add any improvements to the process as you go - consolidating steps, etc, are all very welcomed

rancid birch
upper venture
#

there's the improvements pr for that

#

although, of course, the dream is to have releaser/main.go as the "release entrypoint" - you call a function in there, and it all just goes

#

anyways, i'm on my way out the door now, good luck with the release ❤️

fierce heron
elder wave
fierce heron
#

was just about to rebase

elder wave
#

Tests are passing locally, PR is rebased and approved. So it's just a matter of checks passing.

elder wave
fierce heron
#

❤️‍🔥

#

grep is failing me here, i got timeout failing checks for -q check and engine lint, which gha runs those so i can rerun? @rancid birch @upper venture

rancid birch
fierce heron
rancid birch
#

Oh just saw you merged already, that's okay because it will re-run on main. Still worth a flake issue (whenever you have a sec, not an emergency)

upper venture
#

@rancid birch probably not for this release, but I think I fixed the other remaining cross compilation flake in a PR somewhere

#

I think its the dumb-init one

fierce heron
#

apart from the lint ones obviously

#

i probably had a different PR open in a tab lolsob

rancid birch
#

No problem at all, no harm done. But yeah I've learned over time that when doing a release I have to have a dedicated browser window + terminal and put everything else out of sight in a different workspace otherwise I get lost in my 10k tabs 😄

fierce heron
#

i've got the dedicated terminal, that part was immediately obvious from the exports in RELEASING.md

#

i've been seeing the ✘ moby.filesync.v1.FileSync/DiffCopy X.Xs \n! transport is closing shit randomly while developing, too, so very much some real flaky behavior. any theories?

rancid birch
upper venture
#

Hm somehow the publish workflow failed

rancid birch
upper venture
#

Not seen that one before 😭 maybe the goreleaser bump was bad?

rancid birch
fierce heron
upper venture
#

Yeah I'm actually not able to look, I'm on mobile and should not be looking at this 👀

fierce heron
#

i thought this was suspicious

#

it's only supposed to run on tags, i imagine?

upper venture
fierce heron
#

gotcha

upper venture
#

It looooooks like the release on gh is good

#

The publish metadata step didn't run tho

upper venture
#

Found this

fierce heron
#

sooo what are next steps here then? should i try to make publish more reentrant (eg ignore failures due to the upload existing?)

#

or just rerun maybe?

rancid birch
#

Can we manually run the publish metadata step? Otherwise I guess we might need to delete the published artifacts and re-run the whole job in CI

rancid birch
upper venture
#

I would suggest running them all manually?

upper venture
fierce heron
# 
        uses: ./.github/actions/call
        with:
          function: |-
            cli \
            publish-metadata \
            --aws-access-key-id=env:AWS_ACCESS_KEY_ID \
            --aws-secret-access-key=env:AWS_SECRET_ACCESS_KEY \
            --aws-region="$AWS_REGION" \
            --aws-bucket="$AWS_BUCKET" \
            --aws-cloudfront-distribution="$AWS_CLOUDFRONT_DISTRIBUTION"
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.RELEASE_AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.RELEASE_AWS_SECRET_ACCESS_KEY }}
          AWS_REGION: ${{ vars.RELEASE_AWS_REGION }}
          AWS_BUCKET: ${{ vars.RELEASE_AWS_BUCKET }}
          AWS_CLOUDFRONT_DISTRIBUTION: ${{ vars.RELEASE_AWS_CLOUDFRONT_DISTRIBUTION }}
      # TODO: move this into dagger function call
      - name: "Notify"
        uses: ./.github/actions/notify
        if: github.ref_name != 'main'
        with:
          message: "🚙 Engine + 🚗 CLI: https://github.com/${{ github.repository }}/releases/tag/${{ github.ref_name }}"
          discord-webhook: ${{ secrets.NEW_RELEASE_DISCORD_WEBHOOK }}

lotsa creds here to find lol

upper venture
#

Theoretically. You could delete the release and run it again.

#

This should be "fine"

#

Whether that's good practice or not lol

#

But if it's just ephemeral

#

Then it should succeed

#

(we have done this before I think)

fierce heron
#

You could delete the release
we're talking just the github release here, right? no other side-effect uploads happening here?

upper venture
upper venture
#

So it would push a new engine tag

upper venture
rancid birch
#

I'm checking if I have access

#

manually running the publish metadata seems less risky if possible

fierce heron
upper venture
#

Oh I wonder if - did publish succeed on main before you tagged?

#

That would maybe explain it running twice 🤔🤔🤔🤔🤔

fierce heron
#

When you have confirmed that all checks on $RELEASE_BRANCH are green, run the following: ahhh i read this as PR_BRANCH

#

oh nm it failed finally

rancid birch
#

(double checking if I found the right creds in DMs)

upper venture
#

Yeah okay this will have been the cause

#

The checks effectively ran twice - this is a horrible bug in how goreleaser handles tags

#

I can fix this tomorrow I think

#

But if we deleted the tag and the release and went again, it should just all resolve

upper venture
rancid birch
fierce heron
#

i will fixup the release docs, also. ``` When you have confirmed that all checks on $RELEASE_BRANCH are green, run the following:

#

the when implied that i had done that in previous steps

fierce heron
rancid birch
#

Okay, I was super confused at first because I was looking at the CI runs for the latest commit on main, but for some reason GH links to the v0.15.1 tag CI there, not main. I get what happened a little more now.

I am not sure about being able to just run the publish metadata step manually anymore, there's a bunch of other publish jobs that depend on it that we'd need to run manually too. I'm trying to trace through everything that happened in publish to see what we need to "undo"

#

grabbing a snack quick, I'll join lounge after and look through that

fierce heron
#

sg i just reheated lunch

upper venture
#

Yeah pretty sure about this 🙂

#

I have done this before (at someeee point)

#

So delete release and re-run on tag

rancid birch
upper venture
#

So "it's fine"

rancid birch
upper venture
#

The other super super backup option is "skip .1, it's borked, and do .2"

rancid birch
upper venture
#

It's idempotent-ish yes 😂😂😂

#

A statement

fierce heron
upper venture
#

Yeah that should be fine

#

They're "separate"

fierce heron
#

onnit then

upper venture
#

Except in this bizarre edge case

fierce heron
#

lookin good now

rancid birch
fierce heron
#

was just posting that 🙂

rancid birch
#

That's fortunately not that devastating, easier to do manually

#

everything else looks good though, phew

fierce heron
rancid birch
#

That release token is in our 1password I believe, I can DM you the name. It looks like it should be safe to just run that dagger call locally then

upper venture
#

Nice well done me

fierce heron
#

Yeah definitely string interpolation, although I’m rusty enough with the gha that it’s unclear to me why it’s not templating that particular string

#

oh lul it's missing $

#

you can't forget to give microsoft their $

rancid birch
#

Yeah we can fix in the improve releasing PR

fierce heron
#

i got the thing all prepped to run locally, i just don't know what to actually use as the github.actor value

#

is it me or a bot?

#

(the bot would be the 1p entry's account presumably)

#

found it, it's me

#

hmmm now it's missing a command when trying to run the engine

│ ✔ .withExec(args: ["--addr", "tcp://0.0.0.0:1234", "--addr", "unix:///var/run/buildkit/buildkitd.sock", "--network-cidr", "10.12.34.0/24"], insecureRootCapabilities: true, useEntrypoint: true): Container! 0.0s
│ ✘ .asService: Service! 0.0s
│ ! no command has been set
rancid birch
fierce heron
#

threw a UseEntrypoint on there, but will do withdefaultargs instead when PRing

#

ah nm i think its hanging with the useentrypoint

upper venture
#

(hmmm so super non urgent, but just noticed that we didn't get notifications in the #releases channels for the SDKs)

#

Huuuuuge kudos for taking the release @fierce heron 💜

gleaming hearth
fierce heron
#

i don't trust discord at all today though so 🤷‍♂️

#

on the Double-check that all the above packages have been correctly published and updated to their latest versions. step, confirmed everything except the CLI download, for me the given url https://dl.dagger.io 403s?

rancid birch
fierce heron
fierce heron
rancid birch
#

Hmmm... It works for me on linux/arm64:

sipsma@dagger_dev:~/repo/github.com/sipsma/dagger$ curl -fsS https://dl.dagger.io/dagger/install.sh | BIN_DIR=~/bin sh
sh debug downloading files into /tmp/tmp.DlSpP0mDQk
sh debug http_download https://dl.dagger.io/dagger/releases/0.15.1/dagger_v0.15.1_linux_arm64.tar.gz
sh debug http_download https://dl.dagger.io/dagger/releases/0.15.1/checksums.txt
sh debug display shell completion instructions
#

I'll see if I can check if it's x86 specific somehow

fierce heron
#

works fine on macos too

rancid birch
#

but there are a bunch of provisioning tests that should have caught this earlier, they passed though, so not sure what changed

upper venture
#

Hmmmm oh I wonder if this is because of the fact we deleted some artifacts and we got different checksums

fierce heron
#

sounds likely

upper venture
#

If so it's cached in cloud front

#

The solution would be to invalidate the cloud front cache - I thiiiink the AWS creds we have can do this

rancid birch
#

Yeah if I change install.sh to force x86_64 I hit it too:

sipsma@dagger_dev:/tmp$ BINDIR=./bin ./install.sh 
./install.sh debug downloading files into /tmp/tmp.KdZIxXDidt
./install.sh debug http_download https://dl.dagger.io/dagger/releases/0.15.1/dagger_v0.15.1_linux_amd64.tar.gz
./install.sh debug http_download https://dl.dagger.io/dagger/releases/0.15.1/checksums.txt
./install.sh err hash_sha256_verify checksum for '/tmp/tmp.KdZIxXDidt/dagger_v0.15.1_linux_amd64.tar.gz' did not verify 5230bfd932ecaa02266efd830c5cf793367433242aec2ff6b3c133238bf8d266 vs 228c046ecaaa5a1cb5bc8dea12b8341f34e1128cc834f2dcd921bcbe6e72d10c
upper venture
fierce heron
#

the one extra weird thing here is implication that macos and arm builds are repeatable but the x86 isn't lmao

fierce heron
#

hmm i see ``` WithExec([]string{"aws", "cloudfront", "create-invalidation", "--distribution-id", awsCloudfrontDistribution, "--paths", "/dagger/install.sh", "/dagger/install.ps1"})

// update version pointers (only on proper releases)
if version := cli.Dagger.Version; semver.IsValid(version) && semver.Prerelease(version) == "" {
    cpOpts := dagger.ContainerWithExecOpts{
        Stdin: strings.TrimPrefix(version, "v"),
    }
    ctr = ctr.
        WithExec([]string{"aws", "s3", "cp", "-", s3Path(awsBucket, "dagger/latest_version")}, cpOpts).
        WithExec([]string{"aws", "s3", "cp", "-", s3Path(awsBucket, "dagger/versions/latest")}, cpOpts).
        WithExec([]string{"aws", "s3", "cp", "-", s3Path(awsBucket, "dagger/versions/%s", strings.TrimPrefix(semver.MajorMinor(version), "v"))}, cpOpts)
}
#

WithExec([]string{"aws", "cloudfront", "create-invalidation", "--distribution-id", awsCloudfrontDistribution, "--paths", "/dagger/install.sh", "/dagger/install.ps1"})
is the only one that's invalidation specific

#

so i'll try that i guess

#

i don't think i have these particular AWS powers @rancid birch

#

logged in, found the distribution in CI output, but command is accessdenied

rancid birch
fierce heron
#

RELEASING.md tells me to ping @silk token @urban bison @slate quiver @whole fossil if anything on this PR fails. i don't want to do this but markdown may as well be legally binding. it's just the preview environment creation, no idea if that matters

fierce heron
silk token
silk token
silk token
fierce heron
#

we're all good at this point, hanging shit for tomorrow is https://github.com/dagger/dagger/pull/9189 and https://github.com/dagger/dagger/pull/9192

GitHub
Improve releasing during v0.15.1 by cwlbraa · Pull Request #9189 · ...

mostly generated from following https://github.com/dagger/dagger/blob/main/RELEASING.md, but also:

fixed the WithDefaultArgs breaking change's affect on the daggerverse module's bu...

GitHub
docs: bump dagger-for-github to v7 by cwlbraa · Pull Request #9192 ...

following RELEASING.md, this looks like it might've been skipped previously.

upper venture
#

the problem is that we were auto-detecting which tag to publish under from the git status

#

but - this really really isn't a good valid thing to try doing

#

since we can have the exact scenario that we did - so instead we need to use the tag from the github event

#

🚀 v0.15.1

upper venture
#

but because of the version module, it's all kind of gotten spread out through the code, instead of passed down from one place

#

i wonder if this is just fundamentally the wrong way of doing things (because it's just so stateful)

#

it feels somewhat fundamental - a release is "triggered" by an event (like a main or a tag push) - the type of that event (main or tag) determines what version number we get, but we're currently only attempting to derive this from inspecting the git directory - which might change on re-runs (new tags can appear/disappear)

#

i think what i really really want is the idea of a "contextual event" - so that i could have the version module know what github event triggered it - and then on tagged builds, it just validates that the tag exists, but on main builds, it can auto-derive it's weird pseudo-version vX.Y.Z-timestamp-commit

#

but now that's putting provider specific logic deep into modules, which feels like a design smell

whole fossil
upper venture
#

@fierce heron any specific ideas here, or am i just making no sense?

#

it feels tangential to what we discussed around native ci, and a "event-driven" vs a "state-driven" model

whole fossil
#

Nicely done @fierce heron ! What a first ride 🎠

fast stag
upper venture
#

maybe? it's sort of useful, but i think i'm more interested in what this might look like if we had automatically propagated

#

having thought about my little rant above, I actually think it's okay to determine the version number based off the git state - as long as this doesn't determine the publishing
events should drive publishing (or at least, that's the way we currently do it). the builds can be driven by the state though

fierce heron
fierce heron
#

@upper venture "events should drive publishing" what events are we talking about here?

urban bison
fierce heron
#

"artifact driven" "event driven" "state driven"... all of these things are isomorphic to me depending on context and tools, anyone care to disambiguate and concretize the imagined differences?

#

part of my difficulty here is probably concourse-brain, like in concourse eventing is driven by state changes to artifacts lol

urban bison
#

yeah I can try 😂 these might be slightly different answers depending on who you're asking, but specific to how releases happen:

  • "event driven" - something happens in git like a commit or a tag and we run some jobs. If its a tag, we know what tag was created from the event metadata and can pass that information through for things like build metadata
  • "state driven" - something happens in git like a commit or a tag and we run some jobs. We don't know anything about the event, but we know we were asked to run a release pipeline, so we use the information available in git to decide what to do
  • "artifact driven" - every git commit on main produces artifacts. The process of releasing selects the artifacts of a commit to promote rather than rebuilding everything when we push a tag that we've theoretically already built before
fierce heron
#

❤️ tq that helps, now im curious if @upper venture was using these words the same way. artifact-driven seems a good goal. driving the artifact builds obviously requires some level of state-driven-ness.

from my one run of this, i can say part of the challenge here is that we have a good amount of state to modify when we prep the release --- we commit this, and then need to use artifacts that come from that specific commit, which we eventually tag. this muddles the state vs artifact-driven thing, like a lot of the benefits and simplicity of the artifact-driven approach are lost when we need make changes prior to cutting a release

GitHub
chore: prep for v0.15.1 (#9183) · dagger/dagger@196f232
  • chore: bump dependencies to v0.15.1

Signed-off-by: Connor Braa <connor@dagger.io>

  • chore: add release notes for v0.15.1

Signed-off-by: Connor Braa <connor@dagger.io...

urban bison
#

Yeah exactly, stuff like that would have to change a bit. Basically any commit should be considered release-able (without rebuilding w modified metadata or anything), so bumping any version information would usually happen immediately after a release (so we release 0.15.1, then change immediately main's semver to 0.15.2 so that any subsequent commit could be released as 0.15.2)

fierce heron
#

that makes bumping minors and majors kinda weird, no? i guess maybe you just never release the patch version

urban bison
#

and then any artifact built with this could be released and tagged as 1.19.0-beta1. I think in that case we had a workflow that would go bump that file after a release automatically

fierce heron
#

i just mean that by default you'd release a bunch of 0.15.3-betaXs, then when you decide to bump minors or majors, you'd change that file giving you 0.16.0-betaXs

#

and doing that at the last minute could be weird

#

but i guess worst case the commit that bumps the version would get beta1 and you'd use those artifacts for the release

urban bison
#

yeah it helps to have an automatic process to bump that as soon as you release a version. But there's other ways to solve that problem too