demo-v2

[maiden orbit]

@viscid tartan thanks for deploy-simple! I'm doing some cleanup on the branch, removing the build workflows and figuring out directory naming

[viscid tartan]

sounds good, I started the hard one in JS last night but did not go far, still need to port the netlify cli shellout in JS, let me know if you still need it.

(I am back btw)

[maiden orbit]

Yes deploy-hard still needed

We can work on it together if you want

Otherwise I'll go over everything with new project-centric workflow ( @severe atlas 's last commit) and then try to start the todoapp extension

@severe atlas what modifications are in your netlify extension?

[severe atlas]

Just a change to the cloak.yaml format

I switched to using yaml.UnmarshalStrict yesterday to alleiviate that problem we ran into yesterday, so it's now important that we have that version

[maiden orbit]

Ah ok

[severe atlas]

btw, I mentioned this earlier, but after skimming through my PR again, I'd be fine with merging everything except for one particularly egregious hack (which itself is not too bad to fix), so it should be possible to get rid of my branches from the demo by monday-ish (depending on timing of everything else of course)

[maiden orbit]

I guess the switch to project-centric recursively affects all dependencies?

I guess they're already setup such that 1 extension = 1 project

[viscid tartan]

working on a the "deploy hard" in JS, I just need to add the netlify deploy. Should be done soon...

[maiden orbit]

@viscid tartan FYI I'm renaming them to deploy-1, deploy-2, deploy-3 to avoid confusion

So we have deploy-2 and deploy-3, you're now working on deploy-1

I'll push shortly

Also may I propose @viscid tartan @severe atlas that in each workflow directory, we maintain a README.md with the various commands to run

(I'm thinking of the go workflow in particular)

[maiden orbit]

That would be nice. But just in case, even if you merge, please don't delete the workflow-clean branch 🙂

In fact could we create a demov2 tag on the main repo, and switch to using that everywhere? That way we insulate ourselves from PR movement

[severe atlas]

I was just running through adding a go extension to todoapp to double check and discovered the fun fact that gqlgen, which we need for extension codegen, does some go.mod magic and apparently breaks in the presence of a vendordirectory (???!!!!). Didn't encounter this before because cloak repo doesn't have a vendor...

The vendor dir was needed in order to do builds inside containers, but there's another path. Gonna fix that first so you don't end up blocked

In fact could we create a demov2 tag on the main repo, and switch to using that everywhere? That way we insulate ourselves from PR movement
Sounds good, appending to todo list 🙂

[maiden orbit]

I'm getting an error on yarn run deploy-2 (deploy the easy way ,in JS)

Can't find the actual useful part in the error....

This is where I wish I could share a dagger cloud link 🙂

The error is in the netlify part

                        ^

ClientError: process "/entrypoint" did not complete successfully: exit code: 1: {"response":{"data":null,"errors":[{"message":"process \"/entrypoint\" did not complete successfully: exit code: 1","locations":[{"line":3,"column":5}],"path":["netlify","deploy"]}],"status":200,"headers":{}},"request":{"query":"{\n  netlify {\n    deploy(\n      contents: \"eyJkZWYiOlsiR2k0S0xHUnZZMnRsY2kxcGJXRm5aVG92TDJSdlkydGxjaTVwYnk5c2FXSnlZWEo1TDJGc2NHbHVaVG96TGpFMVVnNEtCV0Z5YlRZMEVnVnNhVzUxZUZvQSIsIkduOEtmV1J2WTJ0bGNpMXBiV0ZuWlRvdkwyUnZZMnRsY2k1cGJ5OXNhV0p5WVhKNUwyZHZiR0Z1WnpveExqRTRMakl0WVd4d2FXNWxRSE5vWVRJMU5qbzBOemsxWXpWa01qRm1NREZsTURjM056Y3dOMkZrWVRBeU5EQTRaR1ZpWlRjM1ptVXpNVGcwT0dKbE9UZGpaamxtWVRoaE1UUTJNbVJoTnpoa09UUTVVZzRLQldGeWJUWTBFZ1ZzYVc1MWVGb0EiLCJDa2tLUjNOb1lUSTFOam80WVdRMk56azNZbUprWVRWbE56Sm1OVGRpT1RVMU5UUmxPVFkyTW1ReE16Wm1ObVkwWlRkbE1HTTRNemRqWXpWbU5tWmxNelJrWWpnNU16RXdObVV6SWlVU0l4RC8vLy8vLy8vLy8vOEJNaFlLQkM5emNtTVE3UU1ZQVNqLy8vLy8vLy8vLy84QlVnNEtCV0Z5YlRZMEVnVnNhVzUxZUZvQSIsIkNra0tSM05vWVRJMU5qbzRZMk0wTVRNd09HUTJPV1U1TVdZelpHUTJPRGRqWWpFM1lUa3pPREV3TmpSaFpEaGlPR1pqT

Oh I see it:

#44 [netlify deploy]
#44 1.037 unexpected error: failed to list sites: [GET /sites][401] listSites default  &{Code:0 Message:}
/Users/shykes/dev/dagger/todoapp/node_modules/graphql-request/dist/index.js:359
                        throw new types_1.ClientError(__assign(__assign({}, errorResult), { status: response.status, headers: response.headers }), { query: query, variables: variables });
                              ^

[severe atlas]

Oh looks like an auth error maybe?

[maiden orbit]

Yeah that's what it is almost certain

[severe atlas]

but yeah, error messages are disgusting right now

[maiden orbit]

I tried to catch it in the script, to have a more readable error. But failed to do so apparently

My attempt:

const netlifyTokenCleartext = process.env.NETLIFY_AUTH_TOKEN
    const netlifyTokenSecret = await client.request(gql`
    {
        core {
            addSecret(plaintext: "${netlifyTokenCleartext}")
        }
    }
    `).then((result) => result.core.addSecret)

I was hoping the first line would fail if the token isn't set

But apparently it doesn't

[viscid tartan]

1. check you have the token exported in your env (NETLIFY_AUTH_TOKEN)
2. if you used a different token that the one I gave you yesterday, you should change the site name

[maiden orbit]

Ah I got it to work 🙂 Now it fails with a nice clear error, and right away (doesn't wait for yarn to complete)

% yarn run deploy-2
yarn run v1.22.19
$ node workflows/deploy-2/index.mjs
Missing netlify API token. Please set it to the env variable $NETLIFY_AUTH_TOKEN
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

I just pushed my cleaned up version

Jumping into a call, will resume after

[severe atlas]

Pushed fixes for go extension codegen and was able to add an extension to todoapp + invoke it with cloak do, so you should be good to go when you get there

[viscid tartan]

pushed deploy-1 to the todoapp / cloak-demo-v2 branch, I ported the ts version from Erik that shells out to the netlify cli

however I realize this sucks because since it's embedded, that netlify shellout runs on the host

I still pushed it because it works, but I am going to re-implem using core gql queries so all of this happens in a container as well

basically removing the dependency on the host, it makes more sense for the demo I believe...

[severe atlas]

Ah right, that only makes sense in a container... yeah switching to shelling out to an execop instead of an actual fork/exec sounds good

[viscid tartan]

yep, almost there

@severe atlas does exec support passing env var?

I don't see it in the gql explorer

[severe atlas]

Not yet, should only take a few minutes to add though

[viscid tartan]

that'd be amazing

because I don't see another way to pass the netlify token to the containerized cli

I am using cloak from your workflow-clean branch btw, if it's not too hard to add it there

I can use it right away

[severe atlas]

Ah okay, yeah we want secret env support for that specifically then... I'll try to add that but if it's hard for some reason I'll just do plain env and can fix later.

I'm guessing you could also pass it by using args like ["sh", "-c", "NETLIFY_AUTH_TOKEN=${foo} netlify deploy"]

But I'll give it a shot either way and push to my branch

[viscid tartan]

I thought about that but the image I use has an entrypoint...

[severe atlas]

Ah okay, yeah forget that then

[viscid tartan]

so I don't specify the binary, just the args...

but honestly clear-text env will do it just fine for now

[maiden orbit]

back from call

[severe atlas]

https://github.com/sipsma/cloak/commit/918be40ed326405f42870e272ff20b1d31dcdc5b Added both env and secretEnv, they were pretty much equally easy

[viscid tartan]

neat, trying it now!

[maiden orbit]

Do you think adding a cache mount would speed up yarn build?

[severe atlas]

yes, I have that on my list of things to do 🙂

First though, I have the switch from sources->extensions+workflows done, but I realized it will actually break you in place because it changes the yarn/netlify extension yamls, so are you good with me pushing that now along side the fixes to todoapp's cloak.yaml?

Have it all ready

[maiden orbit]

yeah sure

[maiden orbit]

🙂

Oh that's a different switch, nvm

[severe atlas]

Ah yeah this is just an issue because of the strict yaml unmarshalling now. Pushed to my branch and to todoapp now though, should be good once you pull

[maiden orbit]

@viscid tartan I get this when running deploy-1 fresh from checkout:

% yarn run deploy-1
yarn run v1.22.19
$ node workflows/deploy-1/index.mjs
node:internal/errors:477
    ErrorCaptureStackTrace(err);
    ^

Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'netlify' imported from /Users/shykes/dev/dagger/todoapp/workflows/deploy-1/index.mjs
    at new NodeError (node:internal/errors:387:5)
    at packageResolve (node:internal/modules/esm/resolve:909:9)
    at moduleResolve (node:internal/modules/esm/resolve:958:20)
    at defaultResolve (node:internal/modules/esm/resolve:1173:11)
    at nextResolve (node:internal/modules/esm/loader:173:28)
    at ESMLoader.resolve (node:internal/modules/esm/loader:852:30)
    at ESMLoader.getModuleJob (node:internal/modules/esm/loader:439:18)
    at ModuleWrap.<anonymous> (node:internal/modules/esm/module_job:76:40)
    at link (node:internal/modules/esm/module_job:75:36) {
  code: 'ERR_MODULE_NOT_FOUND'
}

Node.js v18.7.0
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

[viscid tartan]

because you need to re-do a 'yarn install' , I changed the package.json

however if you wait 5 min, you won't have to

because I just removed it

I am almost there

[maiden orbit]

https://tenor.com/view/im-coming-on-the-way-otw-car-driving-gif-17577492

[severe atlas]

My last push to todoapp had the wrong branch name in cloak.yaml, pushed a fix in case you already pulled.

I'll add cache mount support quick and then go switch to using a tag in our main repo...

[viscid tartan]

if a command fails in a container, it's not possible to get stderr I guess?

my netlify deploy fails, not sure why

[severe atlas]

Its stderr will be printed to the progress output, but yeah I guess it probably is not returned to the client. It actually is probably possible to implement that in the future

[maiden orbit]

If there's any low hanging fruits to making errors more readable, we should pick them before wednesday 🙂

[severe atlas]

In theory the error for netlify should be printed somewhere there

[maiden orbit]

(assuming there's nothing more urgent to do first of course)

[severe atlas]

Yeah I'll think about it. Just limiting the length of the filesystem IDs printed would be a significant improvement

[maiden orbit]

Indeed

@severe atlas should I pull cloak from the branch workflow-test or workflow-clean?

[severe atlas]

clean, I accidentally pushed the cloak.yaml with -test, but fixed it already

[maiden orbit]

Ah ok

@viscid tartan I'm getting the same problem with yarn run deploy-2

Which doesn't make sense since my code is perfect

[severe atlas]

Pushed cache mount support, seemed to work when testing in playground: https://github.com/sipsma/cloak/commit/544fe24fc74dd53f48262d0ad17b2c01e3b018ea

Between that and env var support (to configure YARN_CACHE_FOLDER), we should be good to setup caching now.

Actually I'll go make that change to the cloak yarn extension really quick too

Pushed that too, yarn should be much faster next time you run

[viscid tartan]

ok I found my error, I think it's because of the arch mismatch between the image and my host (amd64 vs arm64)

I found an other image, I'll push once it works, then I'll create a smaller one for arm64

I guess we'll have to port the platform dance we have on dagger 0.2

[maiden orbit]

I just thought of a fun stretch goal for the demo, to make the scenario more realistic

[viscid tartan]

@maiden orbit pushed deploy-1

will do a better netlify-cli container to make it faster to run, the image I use is too fat, but it works...

[maiden orbit]

What's the current command I need to run to generate the go workflow (deploy-3) ?

[severe atlas]

Can you make an image where you run yarn add netlify-cli? Or is that too fat

[viscid tartan]

yes basically what I need, for arm64 and amd64 just to be safe

[viscid tartan]

yesterday it was cloak generate -p workflows/deploy-1/cloak.yaml --sdk=go --output-dir=workflows/deploy-1/ --client

but not sure if that changed

[maiden orbit]

why not just start from alpine like the dagger 0.2 universe package?

[severe atlas]

If you are starting from scratch (i.e. during the actual demo):

1. go mod init github.com/dagger/todoapp
2. cloak generate --output-dir workflows/deploy-3 --workflow

[maiden orbit]

How do you figure out the SDK? look up the output dir in the list of sources in project file?

[severe atlas]

Yes exactly

[maiden orbit]

At this point we could just make it cloak generate and you generate on all workflows by default

(we can add filtering etc. later if we stick to project-centric)

you get all the information you need from the one file; I need to type less flags; win win 🙂

(the command worked btw, thanks)

🙂

@viscid tartan I'm going to fill in the implementation for deploy-3, just checking that you're not already in the middle of it

[severe atlas]

I need to update the behavior to not overwrite your existing files (i.e. main.go) then, but yeah good idea. I'll try it quick

[maiden orbit]

Ah I just understood what you meant. The "panic(implement me)" came from cloak generate

I thought it was the latest commit

[severe atlas]

Ah yeah, it was super trivial to write a file to help users get started, so I just did it. Same as how extensions work.

[maiden orbit]

Nevermind @viscid tartan I got confused

[viscid tartan]

btw for the demo @maiden orbit -> make sure you're login to the docker hub, given the few dependencies on remote images

I guess your docker cache will be warm, but we never now for the tag lookups

[maiden orbit]

Good point

Will cloak be able to use my docker hub credentials?

[viscid tartan]

ah, right...

[severe atlas]

I don't think buildkit will be setup with the creds. I can add that, but have to remember how that works

I think it's pretty easy, but need to double check

[maiden orbit]

https://tenor.com/view/check-list-bj-novak-taking-down-notes-writing-gif-13757327

adding to the list then

Maybe we should start sharing that list so that @minor furnace (and hopefully others soon) can start popping items off it between now and wednesday

[severe atlas]

Adding sharing the list to the todo list 🙂

(will post it somewhere in a min, just finishing the generate thing first)

[severe atlas]

Pushed this, you should be able to just run cloak generate by itself now and it just runs everything, but skips overwriting main.go if it already exists (and prints to output saying so)

[maiden orbit]

Awesome, thanks

I'm just finishing a cleanup pass on deploy-3/main.go

I found a pattern which I find makes the code more readable, will push in a minute

It relies on the generated types being dev-friendly though

Pushed, let me know what you think

https://github.com/dagger/todoapp/blob/cloak-demo-v2/workflows/deploy-3/main.go

[severe atlas]

Yeah LGTM, it's easier for my eyes to break it up into steps now.

netlify.DeployNetlifyDeploy gives me java AbstractFactoryFactory vibes, but it's not a big deal at all. Will be fixed as we iterate on our codegen stuff.

[maiden orbit]

Yeah that's the one where the type feels weird

I didn't get it right on the first try either. If I didn't get LSP auto-complete and lookup definition I would probably still be looking 😉

[severe atlas]

Yeah the output types are the worst part still. But this approach hides that a little bit more at least.

[maiden orbit]

It doesn't remove the weirdness of having to go spelunking in the response object, but it moves it to a place where it doesn't leak everywhere

[severe atlas]

If we change the type in netlify's schema from Deploy to URLs or something, that type might improve. Adding to my list, which I'm going to put in the description of my PR

[maiden orbit]

Here's my plan for the rest of my afternoon (I don't have much time left)

1. Whatever breaking changes to the project infrastructure itself we want to make on monday, we should make now. I'm thinking:

   • Move todoapp to a non-fork repo
   • Setup that demov2 tag

Then my stretch goal 🙂

2. Move deploy-3 to a fake "api" subdirectory, that wants to deploy the todoapp frontend for purposes of testing its new backend API. The API will be in Go, and they will use magefile (like @ancient echo ) so it will all feel more realistic.

btw @severe atlas you should check out how Mage handles their own version of the "workdir problem"

They have two flags:

  -d <string> 
            directory to read magefiles from (default ".")
 -w <string>
            working directory where magefiles will run (default -d value)

Not sure how much of it translates to us, but it seems like a very similar problem

(code that interacts with your code)

[severe atlas]

Okay I'll just do the tag now.

Not sure how much of it translates to us, but it seems like a very similar problem
As it stands we have 2 flags: --workdir which is no longer named sensibly, needs to be renamed back to --context and -p which points to the cloak file relative to the context

I'm appending "think about this" to my list, I agree it's important to get right. And the current state needs at least renaming if not reworking

[maiden orbit]

@severe atlas tag in the main repo right? Seems simpler

[severe atlas]

Yes exactly

[maiden orbit]

It will be slightly more work for you in the short term (update tag when your branch moves) but less moving parts next week which might save us from disaster

[severe atlas]

Agree completely

[maiden orbit]

I think --workdir is fine actually

[viscid tartan]

@severe atlas is it possible that the ENV directive of an image is ignored on the pulled image?

[maiden orbit]

If we mean it exactly the same as yarn: the actual working directory of the tool, with a helper to change it so you don't have to ( cd ... && cloak)

[severe atlas]

just checked, yes we need a one line change to fix that (appending)

[maiden orbit]

https://tenor.com/view/list-to-do-long-first-list-of-things-gif-4577829

[viscid tartan]

ok I found a workaround, but indeed it would make things a bit cleaner on deploy-1

[severe atlas]

tagged, going to update all the configs to point to that git branch now

[severe atlas]

cool will do, it's completely trivial

[viscid tartan]

@maiden orbit pushed deploy-1, have a look, tell me if it's good

I'll have to run soon

[maiden orbit]

Checking now, thanks

I'll be on a plane all monday, they should have wifi but... you never know 😉

By default on monday we can focus on polish, re-running the whole demo on our machines and checking for issues, updating READMEs etc

@proud knot will be doing the same

Thank you very much for your help guys

[viscid tartan]

I'll let you know for Tuesday, I may be able to come over

[maiden orbit]

OK 🙂

Yay just got deploy-3 to work from mage

[severe atlas]

I am getting

module declares its path as: github.com/sirupsen/logrus
                but was required as: github.com/Sirupsen/logrus

again after changing to use the tag somehow, so may take a few more mins, but will finish tonight (nothing to do til 8)

By default on monday we can focus on polish, re-running the whole demo on our machines and checking for issues, updating READMEs etc
SGTM

[maiden orbit]

Now trying to integrate this pretend API sub-project cleanly...

Not sure if I should keep one cloak.yaml or split it in two.

trying the split first

@viscid tartan looks great. I'll play with it in 30mn + get more familiar on monday on the plane

[viscid tartan]

sounds good

we can just remove the last FIXME once we support the ENV directive in the image (the imageConfig needs to be copied explicitely if I remember?) But this is a P2 at this point, it works

[proud knot]

was going to run through now that things are less frothy 🙂 Are we using a branch in @severe atlas 's fork?

[maiden orbit]

We're switching over to a stable tag

(in progress now)

[severe atlas]

(almost done, just testing)

Okay tags pushed: https://github.com/dagger/cloak/tags

And todoapp cloak-demo-v2 branch updated: https://github.com/dagger/todoapp/commit/ef8bf79dcf69c00095399f7f04cd3fc14a4b327c

The Sirupsen import thing got fixed by rearranging the order in which I call tidy and get 😵‍💫

[severe atlas]

https://tenor.com/view/monkey-typing-keyboard-busy-gif-16053597

^ me using go modules

[maiden orbit]

@severe atlas update: I can't split api/ into its own cloak project, because it needs its workdir to be the root of the repo. I see monorepo layout headaches in our future 🙂

Including questions on what the todoapp extension has access to, and if so, what's the API for accessing it

[severe atlas]

because it has a dependency on the other local directory which involves ../ ? Or something else

[maiden orbit]

Yes it's a workflow in a subdirectory, that needs to access source code in ../

I guess with arbitrary host filesystem access, that problem gets solved. Or rather, it becomes the workflow developer's problem 🙂

They can create their own .. spagghetti the usual way

[viscid tartan]

disconnecting for the day, @severe atlas thanks again for the expert-level support, you rock! 🙂

[maiden orbit]

for real

[severe atlas]

Of course, I mean this process is resulting in very rapid improvement, so I appreciate everyone dealing with all the hackiness along the way, it's getting everything in the right direction

[maiden orbit]

Nothing like demo-driven development

Which itself is a form of fear-driven development

[severe atlas]

Yes, reminds me of how the engineers that design airplane electronics are among the first humans to go on test rides 🙂

Slightly less extreme in this case

But agree, this process works really well

[severe atlas]

For this, yeah I guess cloak.yaml has to go in the "most recent common ancestor" directory for all the directories needed in the project right now. I think I need to see more examples of monorepos to understand when that will break.

I'll do some googling later

[maiden orbit]

OK I pushed my new "api" sub-project 🙂

there's definitely some unresolved business regarding workdir access and monorepos.

but nothing we can't handle

@severe atlas one small request for the list: cloak generate only works when run at the root of a project. It should support being run in any directory in the project (since it can walk up the directory tree looking for a cloak.yaml). This makes things a bit easier with the api/ part

actually it's not just cloak generate . It's engine initiazlization too

ie right now this fails:

cd api && mage deploy

whereas this works:

mage -d api -w . deploy

FYI @proud knot @hot latch the demo-v2 document in the cloak is currently out-of-date

I will update it on monday

The source of truth is the cloak-demo-v2 branch in https://github.com/dagger/todoapp

The demo is currently 3 steps:

1. Frontend dev deploys with a npm script (written the hard way)
2. Frontend dev deploys with a npm script (written the easy way)
3. API dev deploys with a Magefile (using Go SDK)

Missing bonus step, monday I hope:

4. Deploy logic is moved to an extension, simplifying both the npm scripts and magefile

[severe atlas]

there's definitely some unresolved business regarding workdir access and monorepos.
Agreed, it's also not hard to change the relevent code, so we can experiment around more.

one small request for the list: cloak generate only works when run at the root of a project. It should support being run in any directory in the project (since it can walk up the directory tree looking for a cloak.yaml). This makes things a bit easier with the api/ part
It technically works if you do cloak generate --workdir ../.., but yeah the automatic "find-up" logic would be much better I agree.

[maiden orbit]

I was wrong @severe atlas the problem is not cloak generate but running my workflow

@ancient echo I can confirm that cloak will work great with mage, and actually it's a pretty awesome combination 🙂

[ancient echo]

awesome!

[maiden orbit]

Here's the demo implementation: https://github.com/dagger/todoapp/tree/cloak-demo-v2/api

[proud knot]

@vestal glacier will have some advice around monorepos and what breaks, I'm guessing 🙂

[proud knot]

Giving things a quick try:

I installed mage via Homebrew.

I pulled the dagger/cloak repo and checked out the cloak-demo-v2 tag.
I built cloak per getting started instructions and ran cloak dev.

I pulled dagger/todoapp repo and checked out the cloak-demo-v2 branch.
cd api
mage -w .. deploy
Lots of error messages. Had to set NETLIFY_AUTH_TOKEN env var.
Went to my Netlify account, generated a new token, export'd the env var.

mage -w .. deploy
lots of stuff working...then errors at end. error Command "react-scripts" not found.
Guessing I need to do an npm/yarn install of node packages including react-scripts locally?
cd ..
yarn install

Command: git
Arguments: ls-remote --tags --heads [e-mail]:dagger/cloak.git
Host key verification failed.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.

hmmm.
tried
npm install complained less 😄

cd api
mage -w .. deploy
Still broken
error Can't add "@dagger.io/dagger": invalid package version undefined.

Okay...found this: https://stackoverflow.com/questions/13363553/git-error-host-key-verification-failed-when-connecting-to-remote-repository

ssh-keyscan -t rsa github.com >> ~/.ssh/known_hosts

remove all the yarn and npm lock files

yarn install seems good.

cd api
mage -w .. deploy

bunch working, then errors at the end again

 > /_shim yarn run react-scripts build:
#0 0.263 yarn run v1.22.17
#40 0.308 info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
#40 0.308 error Command "react-scripts" not found.
------
Error: input:4: yarn.script process "/entrypoint" did not complete successfully: exit code: 1

🤔
If I run yarn react-scripts build or yarn run react-scripts build myself, works fine

oh, maybe I need to do this to set up deploy-3
#1012827742481428560 message

and maybe I need to call like this?
#1012827742481428560 message

was using the README.md in api directory.

Ok, so from root of todoapp (cloak-demo-v2 branch)

go mod init github.com/dagger/todoapp

cloak generate --output-dir workflows/deploy-3 --workflow

Error: unknown flag: --output-dir

hmmm...not sure if interface of cloak generate has changed. Don't see --output-dir

todoapp ➤ cloak generate -h                                            git:cloak-demo-v2*
Usage:
  cloak generate [flags]

Flags:
  -h, --help   help for generate

Global Flags:
  -p, --project string   cloak config file
      --workdir string   workdir as passed to workflows

Okay. I'll stop now. Need direction

https://tenor.com/view/esbritpinedo-gabriel-esbritpinedo-staff-team-cofidis2020-cofidis-my-team-gif-16303007

[proud knot]

checking dagger/cloak repo (demov2 tag) for react-scripts just to be sure...okay not there just here in the todoapp repo.

cloak-demo-v2 ➤ find . | grep package.json | xargs grep react-scripts        git:2a7e20d*
cloak-demo-v2 ➤ find . | grep package.json | xargs grep execa                git:2a7e20d*
./sdk/nodejs/dagger/package.json:    "execa": "^6.1.0",
./examples/netlify/ts/package.json:    "@types/execa": "^2.0.0",
./examples/netlify/ts/package.json:    "execa": "^6.1.0",

todoapp ➤ find . | grep package.json | xargs grep react-scripts        git:cloak-demo-v2*
grep: ./node_modules/resolve/test/resolver/malformed_package_json: Is a directory
./node_modules/react-scripts/package.json:  "name": "react-scripts",
./node_modules/react-scripts/package.json:    "directory": "packages/react-scripts"
./node_modules/react-scripts/package.json:    "react-scripts": "./bin/react-scripts.js"
./package.json:    "react-scripts": "3.4.1",
./package.json:    "start": "react-scripts start",
./package.json:    "build": "react-scripts build",
./package.json:    "test": "react-scripts test",
./package.json:    "eject": "react-scripts eject",

I'll try to run the existing deploy-1 and deploy-2 first...

hmmm... cloak do...???

oh wait, I'm being a node developer, right???

so I run yarn run deploy-1 right?

oooh
built fine but didn't deploy because I need my own site name, probs?

 json: { errors: { subdomain: [ 'must be unique' ] } }

ah, looks hardcoded in deploy-1

    const netlifySiteName = "sam-test-cloak-deploy-js"

flipped to this:

const netlifyToken = process.env["NETLIFY_AUTH_TOKEN"]
const netlifyClient = new NetlifyAPI(netlifyToken)
const netlifySiteName = process.env["NETLIFY_SITE_NAME"].  <<<<<

looks like it built site and tried to deploy it

request: {
    query: '{\n' +
      '  core {\n' +
      '    image(ref: "index.docker.io/samalba/netlify-cli:multi-arch") {\n' +
      '      exec(\n' +
      '        input: {args: ["/netlify/node_modules/.bin/netlify", "deploy", "--build", "--site", "922aba1a-6e34-4110-aad5-8b0c475f9c39", "--prod", "--auth", "syIae1wELD3JNewpc80cTdtQqgsOSbikWUZzgnsGVjo"], mounts: [{fs: "eyJkZWYiOlsiR244S2ZXUnZZMnRsY2kxcGJXRm5aVG92TDJSdlkydGxjaTVwYnk5c2FXSnlZWEo1TDJkdmJHRnVaem94TGpFNExqSXRZV3h3YVc1bFFITm9ZVEkxTmpvME56azFZelZrTWpGbU1ERmxNRGMzTnpjd04yRmtZVEF5TkRBNFpHVmlaVGMzWm1Vek1UZzBPR0psT1RkalpqbG1ZVGhoTVRRMk1tUmhOemhrT1RRNVVnNEtCV0Z5YlRZMEVnVnNhVzUxZUZvQSIsIkNra0tSM05vWVRJMU5qbzRZV1EyTnprM1ltSmtZVFZsTnpKbU5UZGlPVFUxTlRSbE9UWTJNbVF4TXpabU5tWTBa
...<tons of base64>
...
Node.js v18.6.0
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Oh! I had created my site on Netlify by linking it to a git repo 😄

Had to unlink from the git repo for deploy to work (used to work before to start with random git repo and overwrite, not anymore it seems)

NICE!
built/deployed clean

hmmm...but app not working there

trying yarn run deploy-2

fails...seems like NETLIFY_SITE_NAME handled well in deploy-2 hmmmm

okay, new netlify site, deployed the built build directory manually, reset NETLIFY_SITE_NAME

The site was working after manual deploy of build but is broken after yarn run deploy-1!

I set my NETLIFY_SITE_NAME to "" hoping that I'll get a fresh site created.

I did, but the app is still broken...
http://hilarious-kulfi-43b492.netlify.app

checking this
https://answers.netlify.com/t/looks-like-youve-followed-a-broken-link-when-viewing-site/7159/12

when I download a zip of the deploy from netlify, it looks like we deploy the whole repo instead of just the build directory!

It needs the index.html in the build dir, of course. Okay, that makes sense. Let me see if I need to set a workdir or something?

deleted by local build directory and re-ran

deploy-1

Got new site, but no build dir in there. Seems to have deployed my local directory, I expected a build dir to be built in a container and deployed or else for the build to be written back to my machine...hmmmm

going to an appt...will look at again later

[severe atlas]

You now can just run cloak generate by itself, no args, in the directory containing cloak.yaml and it will generate everything it can for you. Or cloak -p path/to/cloak.yaml generate if in a different dir.

I think the parameters for cloak generate changed at least 3 times yesterday, so sorry about the rapidly moving target 🙂 Docs need updating still.

For the deploy, if you go to the build subdir of the deployed site link it works: https://hilarious-kulfi-43b492.netlify.app/build/

That's not expected though, should be at root. The problem seems to be that we aren't first calling netlify link before netlify deploy. Testing a fix quick, will push if successful.

[severe atlas]

Just pushed a fix to deploy-1 that results in build/ being the root of the deployment (also included some cache mounts to make yarn faster)

[proud knot]

will try when back home

[proud knot]

back

trying

[maiden orbit]

Hi @proud knot, a few random things:

• react-scripts: command not found I never got that one. I would say either A) the workflow doesn't call yarn install before calling yarn run build (inside the container, should be 2 distinct queries) or B) if cache mounts were newly added to the workflow, it's possible that there's an issue there which results in an empty yarn cache and therefore no installed dependencies

• yarn run deploy-2 definitely worked when I last used it BUT I noticed deploying a 2nd time to the same site sometrimes (always?) fails, with an error related to "create". I think it's the netlify extension that tries to re-create the site every time, and fails to do so when the site exists (but not 100% sure)

• awesome work getting to this point, thank you!

[proud knot]

Pulled todoapp for changes.

Fixed hardcoded sitename again in deploy-1 since I'm not using that account/token

//const netlifySiteName = "sam-test-cloak-deploy-js";
const netlifySiteName = process.env["NETLIFY_SITE_NAME"]

yarn deploy-1
...
looking good

dang!

failed at end

#36 3.182 ────────────────────────────────────────────────────────────────
#36 3.182   Netlify Build Complete
#36 3.182 ────────────────────────────────────────────────────────────────
#36 3.182
#36 3.183 (Netlify Build completed in 20ms)
#36 3.184 Deploy path: /src/build
#36 3.185 Deploying to main site URL...
#36 3.640 - Hashing files...
#36 3.655 ✔ Finished hashing
#36 4.167  ›   Warning:
#36 4.167 {}
#36 4.167
#36 4.195  ›   Error: No files or functions to deploy
/Users/jeremyadams/src/todoapp/node_modules/graphql-request/dist/index.js:359
                        throw new types_1.ClientError(__assign(__assign({}, errorResult), { status: response.status, headers: response.headers }), { query: query, variables: variables });

let me read where I'm supposed to be running this from again...I keep doing from root

should I do from somewhere else? workflows or somewhere?

oh, I set my NETLIFY_SITE_NAME to ""

lemme fix that

todoapp ➤ export NETLIFY_SITE_NAME=wonderful-medovik-f446fa            git:cloak-demo-v2*
todoapp ➤ yarn deploy-1

#36 3.610 ────────────────────────────────────────────────────────────────
#36 3.610   Netlify Build Complete
#36 3.610 ────────────────────────────────────────────────────────────────
#36 3.610
#36 3.611 (Netlify Build completed in 20ms)
#36 3.612 Deploy path: /src/build
#36 3.614 Deploying to main site URL...
#36 4.026 - Hashing files...
#36 4.038 ✔ Finished hashing
#36 4.559  ›   Warning:
#36 4.559 {}
#36 4.559
#36 4.578  ›   Error: No files or functions to deploy
/Users/jeremyadams/src/todoapp/node_modules/graphql-request/dist/index.js:359

seems like this only works when I do a build locally first

like this

yarn build

now I have a build dir in my root

yarn deploy-1

Works!!

so what is being deployed is my local build, not a build that happens in a container. Is that expected?

I'm going to delete my local build dir again and try deploy-2

Fails...but...seems like it's doing the right things

todoapp ➤ yarn deploy-2                                                git:cloak-demo-v2*
yarn run v1.22.19
$ node workflows/deploy-2/index.mjs

        Using netlify site name: "wonderful-medovik-f446fa"

#1 local://.workdir
#1 transferring .workdir: 7.52kB 0.0s done
#1 DONE 0.0s

#2 copy / /
#2 CACHED
...

respects my NETLIFY_SITE_NAME

#34 /_shim yarn install
#34 CACHED

#35 [build 4/6] RUN go mod init github.com/dagger/cloak/shim/cmd
#35 CACHED

#36 [build 6/6] RUN CGO_ENABLED=0 go build -o /_shim -ldflags '-s -d -w' .
#36 CACHED

#37 /_shim apk add -U --no-cache openssh-client
#37 CACHED

#38 [stage-1 1/1] COPY --from=build /_shim /_shim
#38 CACHED

#39 [build 5/6] COPY . .
#39 CACHED

#40 /_shim yarn run build
#40 CACHED

seems to by doing yarn install and yarn build

presumably in containers

but deploy fails

#41 [netlify deploy]
#41 0.824 unexpected error: failed to create site: [POST /sites][422] createSite default  &{Code:0 Message:}
/Users/jeremyadams/src/todoapp/node_modules/graphql-request/dist/index.js:359
                        throw new types_1.ClientError(__assign(__assign({}, errorResult), { status: response.status, headers: response.headers }), { query: query, variables: variables });
                              ^

ClientError: process "/entrypoint" did not complete successfully: exit code: 1:

going to debug a little by running cloak dev in the todoapp and see if the gql queries in deploy-1 all look good

[severe atlas]

Good catch, there was a typo in the code, just pushed a fix so now it deploys the code built in the workflow: https://github.com/dagger/todoapp/commit/4b2dd03c13f93a45a886876a991cead7b8b0feac

[proud knot]

woot!

guessing the hardcoding of site name is part of the "easy hard way" flow?

//const netlifySiteName = "sam-test-cloak-deploy-js";
const netlifySiteName = process.env["NETLIFY_SITE_NAME"]

I change it for me 🙂

or we could make it like it is in deploy-2

const netlifySiteName = null
if (process.env.NETLIFY_SITE_NAME) {
    const netlifySiteName = process.env.NETLIFY_SITE_NAME
    console.log(`
        Using netlify site name: "${netlifySiteName}"
    `)
} else {
    const netlifySiteName = `${process.env.USER}-dagger-todoapp`
    console.log(`
        Netlify site name not specified in $NETLIFY_SITE_NAME.
        Defaulting to "${netlifySiteName}".
    `)
}

[severe atlas]

Yeah I don't have a strong opinion, but hardcoding in the hard way makes sense to me. Whatever works

[proud knot]

right, hard way 😄

ooo new name too 🙂 const netlifySiteName = "test-cloak-netlify-deploy";

hmmm after changes

#2 copy / /
#2 DONE 0.0s

#3 git://github.com/dagger/cloak.git#demov2
#3 0.526 [e-mail]: Permission denied (publickey).
#3 0.548 fatal: Could not read from remote repository.
#3 0.548
#3 0.548 Please make sure you have the correct access rights
#3 0.548 and the repository exists.
#3 ERROR: failed to fetch remote [e-mail]:dagger/cloak.git: exit status 128

guessing this one...

will try with old args

hmmm

no something else

maybe

FetchError: request to http://localhost:8080/ failed, reason: connect ECONNREFUSED ::1:8080

Error: failed to solve: input:5: core.filesystem.loadProject failed to load cache key: failed to fetch remote [e-mail]:dagger/cloak.git: exit status 128

[severe atlas]

Ah yeah I sent another commit to revert that line after I saw I committed it accidentally 🙂

You need ssh agent and SSH_AUTH_SOCK setup

And a key added to the agent that has access to the cloak repo

[proud knot]

got it.

ah, prob had in my other shell window 😄

eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_rsa_jpadams
ssh-keyscan -t rsa github.com >> ~/.ssh/known_hosts

don't know about SSH_AUTH_SOCK

[severe atlas]

That should be taken care of by the eval "$(ssh-agent -s)"

I think the output of $(ssh-agent -s) is export SSH_AUTH_SOCK=/foo

[proud knot]

ah cool!

Since I'm in a new shell window, lost all of that ☝️ and also my env vars lol. Then put in wrong env var name of NETLIFY_TOKEN instead of NETLIFY_AUTH_TOKEN etc. got fun errors 😄

TextHTTPError: Unauthorized
    at parseResponse (file:///Users/jeremyadams/src/todoapp/node_modules/netlify/src/methods/response.js:13:35)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async callMethod (file:///Users/jeremyadams/src/todoapp/node_modules/netlify/src/methods/index.js:32:26)

It's nice how there's more error checking in deploy-2. Shows a maturing process.

🎉

#36 8.743 ✔ Deploy is live!
#36 8.752
#36 8.752 Logs: https://app.netlify.com/sites/wonderful-medovik-f446fa/deploys/630a6b6f8dc0eb34e66d5e92
#36 8.752 Unique Deploy URL: https://630a6b6f8dc0eb34e66d5e92--wonderful-medovik-f446fa.netlify.app
#36 8.752 Website URL: https://wonderful-medovik-f446fa.netlify.app
#36 DONE 8.9s
Netlify deploy URL: http://wonderful-medovik-f446fa.netlify.app

that's deploy-1 ☝️

looking good!

with no pre-built build directory locally 🙂

gonna remove my local node-modules/ and yarn.lock too.

it was NOT happy about that, which would seem to be because of git SSH stuffs, right?

todoapp ➤ yarn run deploy-1                                            git:cloak-demo-v2*
yarn run v1.22.19
$ node workflows/deploy-1/index.mjs
node:internal/errors:477
    ErrorCaptureStackTrace(err);
    ^

Error [ERR_MODULE_NOT_FOUND]: Cannot find package '@dagger.io/dagger' imported from /Users/jeremyadams/src/todoapp/workflows/deploy-1/index.mjs

I'm going to run an errand. Be back after a bit.

[maiden orbit]

No just an artifact of Sam and I working on each in parallel.

We can reconcile as part of polish

[maiden orbit]

nice!

[proud knot]

to test this, I created a dev/ directory and a minimal package.json, then installed just that module and moved the node_modules/ dir to root.
just creates 42 files in there.

todoapp ➤ cat dev/package.json                                         git:cloak-demo-v2*
{
  "devDependencies": {
    "@dagger.io/dagger": "git+ssh://[e-mail]:dagger/cloak.git#demov2-nodejs-sdk"
  }
}

todoapp ➤ yarn --cwd dev install

todoapp ➤ mv dev/node_modules .

todoapp ➤ yarn run deploy-1                                            git:cloak-demo-v2*
yarn run v1.22.19
$ node workflows/deploy-1/index.mjs
node:internal/errors:477
    ErrorCaptureStackTrace(err);
    ^

Error [ERR_MODULE_NOT_FOUND]: Cannot find package 'netlify' imported from /Users/jeremyadams/src/todoapp/workflows/deploy-1/index.mjs

So I'll look at the index.mjs. Seems we just rely on the user running yarn install on their own machine and all of the modules being in the root dir's node_modules/ dir...hmmm, but we run yarn install in the index.mjs so that doesn't make sense...

If I run yarn install on my laptop with just the dev dependencies, the other node modules get added right on top. In total we have 1077 files in there after it runs.

Maybe we're just looking for presence of the node-modules/ dir and bail out of yarn install if already present??? NO I don't see that..we're just running yarn install in the workflow.

I fired up an alpine container with the repo's root dir mounted to /src with the minimal set of 42 files in node_modules but with the full package.json and ran yarn install, no problem in installing the rest of modules over the top.

So seem's we've got some kind yarn install directory issue maybe? I know there is a --cwd option for yarn.

[maiden orbit]

@minor furnace most useful thing you can do is go through the todoapp demo v2 from scratch like @proud knot did, get it working, and fix whatever issues you run into. We’ve been following that process last week and made very rapid progress

[minor furnace]

@maiden orbit @severe atlas Probably too late for the demo, but by using FS chaining, the entire deploy-2 workflow could be written as a single query:

query ($site: String!, $token: Secret!) {
    workdir {
        yarn(run: "build") {
            netlify(dir: "build", site: $site, token: $token) {
                url
                deployURL
            }
        }
    }
}

deploy-2/index.mjs would end up being 20-30 lines total (deal with secrets, some wiring, etc)

[severe atlas]

Yeah I have adding a Filesystem extension to yarn as a stretch goal, but netlify would work too. It’s entirely possible still, shouldn’t take long at all in theory

[minor furnace]

Hey @severe atlas 🙂

Running into a bit of a problem executing the demo and it looks like the stdout/stderr messages are suppressed, somehow. Nothing urgent we can look together later today

All I see is graphql-client having an error, and the entrypoint exited with code 1 but that's it

[severe atlas]

Sometimes it’s just buried in the massive amount of output, not sure if it’s that or actually missing. In my most recent pr updates I switched FSID to be a hash with llb stored serverside (somehow resulted in deleting more lines of code than adding), which helps with the output a lot

[minor furnace]

Nice (FSID hashes)!

No it's not buried, AFAIK

That's what I thought so I scanned the output line by line, unless I missed it I don't think it's that

[minor furnace]

Ah, I thought it would be more complex than that 🙂

using the global sync.Map saved a bunch of headaches

nice trick

[severe atlas]

Ah okay, by heuristic it’s probably that your token and site name don’t align, but yeah the error handling must be off somehow too

[severe atlas]

Yeah and I don’t even feel that much shame using a global var since we already have an issue open to switch to per-extension schema views, which is probably the right time to to deglobalize that too 🙂

[minor furnace]

No shame at all ... if for some reason 2 FS ID end up clashing (same session or different session), we have much bigger problems on our plate 🙂

[severe atlas]

Haha very true!

[severe atlas]

Added Filesystem extensions here, tested and seems to work: https://github.com/dagger/cloak/pull/95/commits/14ce3147dc32af2a8dd2d4dcf8cf732e5f8d3ab5

It's really easy to do in TS right now. In Go it's more annoying because the codegen needs updates, so you'd have to modify the autogenerated code by hand. Once we improve codegen it should be easy there too though

[proud knot]

for a bit of fun, created this Dagger plan to run cloak deploy-1 workflow:
https://gist.github.com/jpadams/5bbf21e154ade1dd49153220fc792917

Also helped me see that even in a containerized enviro, must run yarn install before attempting to run yarn deploy-1 aka node workflows/deploy-1/index.mjs

[proud knot]

In that ☝️ I also cloned the dagger/cloak repo in the todoapp repo to have it available for Dagger via core.#Source

see notes at top of gist

[maiden orbit]

I’ll be on a plane all day today. There will be wifi but I don’t know if ssh will work 🥲

[severe atlas]

I just realized when testing rebases there's a flaw in the change to using a hash for FSID: if a result in cached from a previous run, it can return FSIDs for steps that were skipped during the current execution, in which case we won't know the mapping between the hash and actual LLB. So it sometimes breaks depending on the caching of the run.

No easy fix unfortunately, I actually am wondering if this would be a good upstream change; I can imagine buildkit exposing an API that lets you retrieve cached results based on LLB digest. Either way, will have to revert this for now.

cc @minor furnace ^

[maiden orbit]

Not too late! Do you mind making the change? I didn’t realize it was implemented 🙏

[severe atlas]

Sure, yeah I will update the tag and then give updating todoapp a shot

[maiden orbit]

Ah ok I thought @minor furnace was saying that it’s implemented

I understand now 🙂

[minor furnace]

@maiden orbit what's the state of todoapp? Would you rather we don't make changes directly, let you handle merging/etc?

or do you have a "good commit" on your end?

[maiden orbit]

Of course it would be great to have but it depends on whether you have more urgent thion the list already, prioritize accordingly

[severe atlas]

To be clear, I already made the change to the yarn and netlify extensions in cloak. I just need to update todoapp to actually use the new chaining

[maiden orbit]

@minor furnace you can make changes as long as we communicate early and often on this thread

Especially today I’m flying with limited wifi so won’t be very productive I think (if I can code at all)

[minor furnace]

It wasn't implemented, and now it's on @severe atlas's branch 🙂 I wasn't sure if you wanted to show that DX or go with less chaining

so I held off

[maiden orbit]

in doubt if you see a change that would help, go ahead 🙂

[maiden orbit]

I think chaining is better.

In any case I’ll be happy to just show something that works and isn’t completely confusing

[minor furnace]

there's this one also by @fleet anvil: https://github.com/dagger/todoapp/pull/9

Sets up basic linting etc for JS, good to merge?

[maiden orbit]

We spent the last few days just getting things to work, didn’t have much time for polish

[minor furnace]

sounds good. Today was my first day back, I wasn't sure what was labeled with virtual "DO NOT TOUCH" duck tape 🙂

[maiden orbit]

@minor furnace important context 👆

Adding step 4 would be awesome

The scenario is “frontend dev improves their npm scripts with cloak; backend dev improves their magefile with cloak; then they reduce duplication by moving common logic in an extension; this was the first step towards organically building an internal platform “

also if you want to change to npm, go for it

As long as everything works tonight 🙂 Tomorrow is finishing touches & rehearsal

[proud knot]

I updated my Dagger driver of cloak to allow input of deploy-1 or deploy-2. Just like in my local m1 mackbook enviro, deploy-1 works, but deploy-2 doesn't.
I run it like this:
dagger do --with "actions: yarnScript: \"deploy-1\"" --with "actions: sock64: \"$(echo -n $SSH_AUTH_SOCK | base64)\"" run --log-format plain
https://gist.github.com/jpadams/5bbf21e154ade1dd49153220fc792917
I figure this gives me repeatability while trying to get deploy-2 to work.

(one caveat is that I clone the cloak repo into my todoapp repo before running it. See notes at top of gist)

[proud knot]

@severe atlas @minor furnace not sure how y'all are doing on deploy-2 does it "just work" for you?

This is the error I get
https://gist.github.com/jpadams/245d621954f145c894c225fd794bfde9

[severe atlas]

just been chatting about it, @fleet anvil fixed all the problems it turns out, just merged his stuff and it's working for us again

There were some problems with setting NETLIFY_SITE_NAME

[proud knot]

woah! sweet!

new stuff on that git tag?

[severe atlas]

it was just problems in todoapp repo

[proud knot]

nice

running my dagger driver again with new todoapp code

11:49AM INFO  actions.run._exec | #7 120.7 #47 [netlify deploy url]
11:49AM INFO  actions.run._exec | #7 120.8 #47 DONE 0.1s
11:49AM INFO  actions.run._exec | #7 120.8
11:49AM INFO  actions.run._exec | #7 120.8 #48 [netlify deploy deployURL]
11:49AM INFO  actions.run._exec | #7 120.8 #48 DONE 0.0s
11:49AM INFO  actions.run._exec | #7 120.8
11:49AM INFO  actions.run._exec | #7 120.8 #49 [netlify deploy deployURL]
11:49AM INFO  actions.run._exec | #7 120.8 #49 DONE 0.1s
11:49AM INFO  actions.run._exec | #7 120.8 Netlify deploy URL: http://wonderful-medovik-f446fa.netlify.app
11:49AM INFO  actions.run._exec | #7 120.9 Done in 33.20s.

Looks good!!!!

seems like a lot of yarn installs happen in deploy-2...maybe I don't need to run it myself...will try

[maiden orbit]

About yarn install:

• The cloak pipeline needs to run it (part of a standard npm/yarn buuld)
• As a convenience we made the dagger sdk a dependency on the app so you can install it straight from your familiar environment. That’s the only reason you should need to run yarn install on the host

[proud knot]

will digest that ☝️ but it's true that you have to run yarn install in todoapp to get yarn deploy-1 to work, right?

That's been my experience.

[maiden orbit]

Normally it should only be required to get the dagger.io/dagger package (the sdk)

[proud knot]

Also same in deploy-2. I just disabled running yarn install in my Dagger driver and it broke it

yeah, must be that package

[maiden orbit]

so, yes. but it depends what error you get

if it complains about something else (like react-scripts missing) that’s not normal

[proud knot]

11:56AM INFO  actions.run._exec | #7 46.13 yarn run v1.22.19
11:56AM INFO  actions.run._exec | #7 46.16 $ node workflows/deploy-2/index.mjs
11:56AM INFO  actions.run._exec | #7 46.19 node:internal/errors:477
11:56AM INFO  actions.run._exec | #7 46.19     ErrorCaptureStackTrace(err);
11:56AM INFO  actions.run._exec | #7 46.19     ^
11:56AM INFO  actions.run._exec | #7 46.19
11:56AM INFO  actions.run._exec | #7 46.19 Error [ERR_MODULE_NOT_FOUND]: Cannot find package '@dagger.io/dagger' imported from /src/workflows/deploy-2/index.mjs
11:56AM INFO  actions.run._exec | #7 46.19     at new NodeError (node:internal/errors:387:5)
11:56AM INFO  actions.run._exec | #7 46.19     at packageResolve (node:internal/modules/esm/resolve:951:9)
11:56AM INFO  actions.run._exec | #7 46.19     at moduleResolve (node:internal/modules/esm/resolve:1000:20)
11:56AM INFO  actions.run._exec | #7 46.19     at defaultResolve (node:internal/modules/esm/resolve:1214:11)
11:56AM INFO  actions.run._exec | #7 46.19     at nextResolve (node:internal/modules/esm/loader:165:28)
11:56AM INFO  actions.run._exec | #7 46.19     at ESMLoader.resolve (node:internal/modules/esm/loader:844:30)
11:56AM INFO  actions.run._exec | #7 46.19     at ESMLoader.getModuleJob (node:internal/modules/esm/loader:431:18)
11:56AM INFO  actions.run._exec | #7 46.19     at ModuleWrap.<anonymous> (node:internal/modules/esm/module_job:76:40)
11:56AM INFO  actions.run._exec | #7 46.19     at link (node:internal/modules/esm/module_job:75:36) {
11:56AM INFO  actions.run._exec | #7 46.19   code: 'ERR_MODULE_NOT_FOUND'
11:56AM INFO  actions.run._exec | #7 46.19 }

[maiden orbit]

you know a good thing to try

the API dev running mage, could do so from a host with no yarn or npm installed

then you’ll know

[proud knot]

I provide the SSH_AUTH_SOCK, so it seems that cloak doesn't know how to use that

or, yarn doesn't or something...

not getting mounted in a container

[maiden orbit]

@pastel mesa 👋 this thread is the bleeding edge, we’re working on the new demo of cloak showcasing a realistic adoption scenario and slightly more polished DX

We’re showing it for the first time on wednesday so it’s a bit of a scramble

[minor furnace]

what's up with mage?

[maiden orbit]

👆

We use it in step 3 to show the Go sdk in a realistic setting

it’s pretty neat 🙂

[minor furnace]

Oh

[maiden orbit]

see the api/ dir

[minor furnace]

I actually today did this, dogfooding cloak with cloak using mage: https://github.com/dagger/cloak/pull/136

I could try that step 3

[maiden orbit]

mage is pretty awesome. very complementary to cloak

i’ll give w shoutout in the demo

[minor furnace]

Ha, I was thinking exactly the same thing today

I wanted to dogfood. Started writing some Go. Didn't want to deal with handling a main, arguments, blah blah

mage fits the bill perfectly

[maiden orbit]

yup

I almost made site name an arg but didn’t know how to make it optional

[minor furnace]

I like that with current "workflows" I'm not constrained to 1 way of doing things, I can just integrate whatever tool does the job best

(vs the previous experiment where we had a "cloak do thing" -- it's cool to delegate that "do thing" to other tools, there's plenty)

[maiden orbit]

yes agreed

the raw Go sdk kind of got in the way of explaining that, since it’s pretty rare to replace makefile with go

but showing mage solves that

[minor furnace]

Tonight PST?

it's 9pm here, I'm going to check out soon

[maiden orbit]

Lol I guess tomorrow morning PST 🙂

[minor furnace]

a meta-bonus would be to add tests the native way (e.g. jest and go test). Meta because it would guarantee the demo still works AND it can be demo'ed

[minor furnace]

Oh, got it. I didn't see step 3 -- it's in API

[maiden orbit]

My favorite part is that we can orchestrate those native tests with cloak since we will support arbitrary depth nesting unlike docker

cloak in cloak in cloak in cloak in…

[minor furnace]

Was just chatting with Erik about that. I tried to add a mage test to our own repo, but of course failed because of nested cloak

But it wouldn't be hard to support

And actually dogfooding makes a ton of sense for us: it's going to be a PITA to manage N SDKs in different languages plus cloak itself (lots of different buildchains)

[maiden orbit]

Yes it would be a killer feature

[minor furnace]

we could literally have a mage sdk:publish or whatever to publish all SDKs

[maiden orbit]

not for beginners but all power users will love it

[minor furnace]

or like, mage lint could just use proper for loops to scan all examples and run the linter

no more find | xargs <bunch of crap>

[maiden orbit]

Ok I need to submit a title for this presentation… any ideas?

It’s a lightning talk at Vmware Explore, at Moscone Center

(I presume in an obscure back room of moscone)

[minor furnace]

"Untitled Hack #6"

Forgot which one of the users mentioned a "hackernews worthy title" (his own words), "automate your CI/CD pipeline with graphql"

[maiden orbit]

I’m trying to not make the title itself directly about cloak

for messaging reasons

not

not directly

(taking off, will be back shortly on wifi 🤞🤙)

[minor furnace]

Ok then something about programming ci/cd pipeline with code. Title works with CUE dagger, makes the transition to cloak smooth

[severe atlas]

No specific suggestion, but I like the idea of "building automation platforms". It kind of works with Europa Dagger I suppose, but it especially makes sense w/ cloak where extensions are essentially a way of building+distributing platforms (as graphql APIs). I'm bad at naming things though.

[maiden orbit]

I sent this:

Incremental Revolution! Let’s build our own automation platform in 20mn, one artisanal script at a time

[severe atlas]

Did you also mount the socket in too? You'll need that in addition to the env var

[proud knot]

Dang the mage example just works now that deploy-2 is all squared away! works on my laptop, anyway 😉

[proud knot]

yep, I mount it, but it's only if I leave off the initial yarn install that things fail

the initial one in my dagger driver that has the SSH_AUTH_SOCK mounted.

happy to show you.

[severe atlas]

Ah okay, that actually makes perfect sense, nevermind. You have to have that because its imported in the script running direct on your host. yarn run won't run auto import missing deps, so that's just normal yarn/node things afaik

[proud knot]

yeah, I isolated this in a container (driven by dagger) so I could repeat it without worrying as much about the state of my laptop.

in that process I noticed that with a fresh todoapp repo (with no node_modules yet), I needed to have my automation run yarn install once before attempting the cloak run via yarn deploy-1 or yarn deploy-2. Sounds like that's not surprising. Cool.

Though I kinda thought cloak could do it for me since I see a bunch of yarn install runs happening. If iI could mount the same SSH_AUTH_SOCK that I'm mounting in my dagger automation in cloak, figured it woudl just work to get that pesky @dagger dependency.

[severe atlas]

Though I kinda thought cloak could do it for me since I see a bunch of yarn install runs happening. If iI could mount the same SSH_AUTH_SOCK that I'm mounting in my dagger automation in cloak, figured it woudl just work to get that pesky @dagger dependency.
Those take care of installing the dependencies for builds happening inside cloak. But the thing is you need that dependency to even invoke node workflows/deploy-1/index.mjs at all, even on the host

It's kind of meta and confusing

[proud knot]

ahhhh. okay. that's the one dep I need to bootstrap everything else

[severe atlas]

Yep exactly

[severe atlas]

Got the yarn+netlify chaining working in todoapp, pushed it: https://github.com/dagger/todoapp/commit/c27b7f41872ce3ae24a7f524939880cd57b49912
Workflow is now pretty much just:

  // 1. Load netlify token into a secret
  const netlifyTokenSecret = await client
    .request(
      gql`
        {
          core {
            addSecret(plaintext: "${netlifyTokenCleartext}")
          }
        }
      `
    )
    .then((result) => result.core.addSecret);

  // 2. Build with yarn and deploy to netlify
  const result = await client.request(gql`
    {
      host {
        workdir {
          read {
            yarn(runArgs: ["build"]) {
              netlifyDeploy(
                subdir: "build",
                siteName: "${netlifySiteName}",
                token: "${netlifyTokenSecret}",
              ) {
                url
              }
            }
          }
        }
      }
    }
  `);

Also ran into a bug along the way and realized that one of the reasons we are not always seeing error messages is a race condition: when a workflow gets an error it immediately shutsdown the cloak engine, which may have not yet flushed all the buildkit progress logs (where errors are currently reported). The fix will probably be a hacky "sleep for a second if there's an error before shutting down engine" in the nodejs sdk, but will push it since not always seeing errors is pretty devastating

[severe atlas]

Pushed updates for that ^

It required updating the demov2 tags but apparent yarn cache doesn't like that (it has to pull the nodejs sdk package using git and fails with the error would clobber existing tag) , so you need to run yarn cache clean on your host and also clear out your buildkit cache (docker volume rm dagger-buildkitd, etc.) in order to flush out the yarn cache mount we are using in there now... 😞

[proud knot]

Error: unknown flag: --workdir

[viscid tartan]

@severe atlas I saw you added the 'netlify link' in deploy-1. I omitted it initially to keep the query shorter, and I don't think it's needed since the deploy takes the arg --site. Obviously I don't mind leaving it, just curious to know if you had more reasons to have it here.

[proud knot]

just pulled cloak and rebuilt, pulled todoapp. cleared caches/volumes.

[severe atlas]

When we didn't make that link call, the deployment ended up being the entire source code directory rather than just the build subdir. I didn't really understand why, but changing to use link first fixed it: #1012827742481428560 message

[viscid tartan]

weird, that should be taken care of with the workdir... I am sure the link isn't needed but if that fixes it, let's leave it like that for now

[proud knot]

I'm only one with that error? Error: unknown flag: --workdir hmmm...what have I done 😄

[severe atlas]

Yeah doesn't make sense but oh well for now (my current life motto)

[severe atlas]

Are you building cloak from the demov2 tag?

[proud knot]

yes

[severe atlas]

What command are you running when you get that error? deploy 1/2 or the mage one?

[proud knot]

well...let me just redo everything

looking good.

must have been lost somewhere in the cloak tree 🙂

[severe atlas]

Okay good to hear, no worries

[proud knot]

yarn install locally to get things bootstrapped
deploy-2 just works, perfectly
deploy-1 errors out unless I modify netlifySiteName in the index.mjs (back to that being expected, right?)
cd api; mage -w .. deploy works perfectly...

[severe atlas]

deploy-1 errors out unless I modify netlifySiteName in the index.mjs (back to that being expected, right?)
That's still how it is for me, I didn't think we had fixed it yet

[maiden orbit]

Well there was no wifi on the plane 😅

[severe atlas]

No worries, nothing too dramatic happened since. I was just running through starting with empty dirs and filling things in as will happen during the demo. Going to push a tiny tweak to the mage setup. I just added the feature to cloak where if workdir+configPath are unset, it will look up directories until it finds one. This results in you being able to just run cloak generate from the api or api/magefiles dir directly and everything works automagically.

Without that I was going to have to move go.mod around and stuff, but now it's relatively clean actually.

[severe atlas]

Pushed that now ^ Also nice because you can just do mage deploy now instead of mage -w .. deploy

[maiden orbit]

Awesome, I’ll try it tonight from home. Then early bedtime & big day of prep tomorrow 🙂

[severe atlas]

I had to overwrite the demo2 tag today so FYI you'll need to first clear various caches which all seem to error out or silently not update when that happens:

1. yarn cache clean
2. go clean -modcache
3. The buildkit cache (docker volume rm dagger-buildkitd, etc.)

[maiden orbit]

ok. I always assume tags are mutable, so forget that not everybody does

[severe atlas]

Yeah no one seems to handle that the same way

[maiden orbit]

Maybe I should have suggested a branch instead. sorry

[severe atlas]

Oh no worries at all, who knew this would happen and it's a very minor inconvenience anyways

[severe atlas]

@viscid tartan I looked again and once again was getting the same problem as before if I left out netlify link. But I found that I could leave out link if I added --dir to the netlify deploy command. Updated to that since it's simpler: https://github.com/dagger/todoapp/commit/1f22d8d57ab1537bbcaba395a784691a3299a02a

Also, turns out that even though there is a cool trick you can use to get LLB defaults filled out from an Image with a one-liner, it doesn't work it you send the state through an marshal/unmarshal cycle (not sure if expected or bug, seems weird). So including the image config defaults is more effort than immediately worth and I removed your FIXME comment.

[viscid tartan]

Sounds good, thanks! I’ll try again tonight to make sure it works on my end as well.

[viscid tartan]

it's all good, it works

[viscid tartan]

@severe atlas @proud knot I just pushed a fix, ported from deploy-2, no more netlifySiteName hardcoded.

[maiden orbit]

I'm getting frequent crashes in the graphiql sandbox. Trying to look at the API docs for exec, but it keeps crashing (browser page goes blank). Browser reload brings it back, but it clears the docs navigation (query editor is not affected though)

There's other weird behavior, the contents of some pages keep moving around at what looks like exactly every 1sec. Schema page; docs search result. Like the graphiql frontend polls the API every 1sec and re-renderss the result in a different order

This is with cloak dev from a fresh checkout of the new demo tag (5mn ago)

4f595cceb706b029679119db52370c80e9bd59bb

[severe atlas]

That issue has existed for a while, it’s due to the graphql server library we use. There’s a pr open with them with a fix (from someone else, not us)

[maiden orbit]

OK it's not a P1 for the demo for sure. Good to know in advance though so it doesn't take me by surprise

[severe atlas]

Have not experienced this before. It’s only when looking at the exec docs? Or in general?

[maiden orbit]

Not sure, it's coming and going. Will let you know if I see a more clear pattern

It looks like the "can't deploy to an existing netlify site" issue is solved as well?

[severe atlas]

Yeah everything has been passing consistently for me for a while. I think for @proud knot too last I heard

[maiden orbit]

https://tenor.com/view/kronk-emperors-new-groove-its-all-coming-together-gif-12813012

heads up I'm renaming a few dirs in todoapp for cosmetic / educiational reasons

• deploy-1 -> kick-the-tires ("Demo 1: let's kick the tires by writing a first script")
• deploy-2 -> ("Demo 2: writing a deployment script in JS")

This leads to a cosmetic request @severe atlas (no action required tonight..) I think we should try renaming "workflows" to "scripts" in the DX.

The concept of "artisanal scripts" seems to be sticking well with people, it turns into a natural explanation of what cloak does: incrementally improve your artisanal scripts by backing them with an API so that they don't attract technical debt as they grow.

Using the word "script" where we currently use "workflow" would reinforce that explanation, and make the DX more consistent

Note: I just renamed the workflows/ dir to scripts/ in todoapp for this very reason

[severe atlas]

Yeah script is the exact word that I keep going back to too. My only hesitation has been that it’s sort of weird that you can write “scripts” in languages like Go, which aren’t “scripting languages”, but I’m probably overthinking it. It sounds completely right in the sense that you are scripting extensions, tying them together, etc.

A semi-related thought for post-demo work: I’ve been realizing that the main difference between extensions and workflows (now scripts) are all things controlled by SDKs (schemas, codegen, runtime, etc.). So maybe our yaml config and graphql api doesn’t need any awareness of extensions vs. scripts. Maybe it only knows there’s source directories associated with an SDK. And then there is a “Go extension SDK” and a “Go scripting SDK”. Or maybe there are settings within the Go SDK to control whether you are writing a script or an extension, etc.

Think this could be a route to simplify and not over opinionate our low level API while still being able to create opinions on top of it.

[maiden orbit]

It's really the dynamic duo of Scripts and Extensions that makes it work.

• Scripts are meant in the usual sense that you mean. It's indeed unusual to script with Go. That's what makes Magefiles surprising, and neat.
• Extensions are not scripts - it's not surprising at all to have them written in Go.

So I do think we need awareness of those core concepts. Of course if the internal implementation is 99% the same code I don't have a problem with that 🙂

But I think there will be substantial differences at least in the upper layers of the DX. Things that are superficial to the core implementation but make a real different to devs.

[minor furnace]

I'm having a problem with the demo, the cache never seem to be busted. Making changes to todoapp doesn't result in an update of the netlify site /cc @severe atlas

[minor furnace]

If we go ahead with that hack, we should file an issue so we remember to remove that sleep(1) and do a proper fix

[minor furnace]

Bisected back to this:

commit c27b7f41872ce3ae24a7f524939880cd57b49912 (HEAD)
Author: Erik Sipsma <[e-mail]>
Date:   Mon Aug 29 13:58:49 2022 -0700

    Update to use yarn+netlify Filesystem extensions.

Also, when wiping the cache, I get this error (only once with a fresh cache):

/Users/al/work/todoapp/node_modules/cross-fetch/node_modules/node-fetch/lib/index.js:1491
            reject(new FetchError(`request to ${request.url} failed, reason: ${err.message}`, 'system', err));
                   ^
FetchError: request to http://localhost:8080/ failed, reason: connect ECONNREFUSED ::1:8080
    at ClientRequest.<anonymous> (/Users/al/work/todoapp/node_modules/cross-fetch/node_modules/node-fetch/lib/index.js:1491:11)
    at ClientRequest.emit (node:events:537:28)
    at Socket.socketErrorListener (node:_http_client:465:9)
    at Socket.emit (node:events:537:28)
    at emitErrorNT (node:internal/streams/destroy:151:8)
    at emitErrorCloseNT (node:internal/streams/destroy:116:3)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  type: 'system',
  errno: 'ECONNREFUSED',
  code: 'ECONNREFUSED'
}

No other error message, the previous line before the error is #18 31.96 [4/4] Building fresh packages...

So here's how it goes:

0. Make a change
1. Clear the cache. First run fails
2. Run a second time. It works
3. Run a third time. It's all cached, nothing happens

[fleet anvil]

In my side I've got the following error for deploy-1 and deploy-2:

#32 /_shim yarn install
#32 0.240 yarn install v1.22.19
#32 0.312 [1/4] Resolving packages...
#32 0.667 [2/4] Fetching packages...
2022/08/30 15:58:14 agent 27: ssh: parse error in message type 27
2022/08/30 15:58:15 agent 27: ssh: parse error in message type 27
#32 3.985 error Command failed.
#32 3.985 Exit code: 1
#32 3.985 Command: git
#32 3.985 Arguments: fetch --tags
#32 3.985 Directory: /cache/v6/.tmp/7aaec7a2b05007359408cd8e35622ff3
#32 3.985 Output:
#32 3.985 hostfile_replace_entries: mkstemp: Read-only file system
#32 3.985 update_known_hosts: hostfile_replace_entries failed for /root/.ssh/known_hosts: Read-only file system
#32 3.985 From github.com:dagger/cloak
#32 3.985  ! [rejected]        demov2            -> demov2  (would clobber existing tag)
#32 3.985  ! [rejected]        demov2-nodejs-sdk -> demov2-nodejs-sdk  (would clobber existing tag)

any idea @minor furnace ?

both repo cloak and todoapp are up to date + I've recreated a fresh dagger buildkit container

ok it works now 👍 ! I've deleted my docker volume

[severe atlas]

Glad you figured it out, you may also need to run these other commands if we update the tag again: #1012827742481428560 message

[fleet anvil]

nice! thanks

[severe atlas]

Ah I wonder if the parent result isn't making its way into the cache? But I thought it should. Let me reproduce it and see what's going on

[severe atlas]

I've been leaving FIXME(sipsma) comments and not merging into main for all the egregious stuff like this (there's more than just this sleep). Plan is to either fix before merging or make an issue before merging for all of them

[severe atlas]

Can’t reproduce the connection refused error, but can reproduce that changing todoapp doesn’t bust the cache.

Have a suspicion, even though we provide the fsid (and thus llb) of parent results, the LLB may not have changed even if the filesystem contents themselves have. Simplest solution is to just mount the parent result in if it is a Filesystem, which kicks in buildkits content based hashing. Also not bad in the sense that it then matches the behavior of resolver args. Will give it a shot to see if that’s right and go from there

[severe atlas]

Yep that seems to fix the issue for me, changing todoapp invalidates the cache and results in redeployment now: https://github.com/dagger/cloak/pull/95/commits/0d072607b2b02f1799ea978ca1090c4e96c25fb0

[minor furnace]

ugh 😐 makes sense

but yeah makes total sense

[severe atlas]

Once again, far from an elegant solution, but in the longer term it probably does make sense for there to be an option to mount in the parent result if it's a filesystem (same as resolver args) and if we want this same behavior but without mounting then we will have to get the content hash (I think this can be obtained from buildkit, don't have to reimplement it ourselves) and just include it with something in the execop, such as /dagger/inputs.json

[minor furnace]

I'm thinking the same problem might be laying around in other spots

whether your parent is a Filesystem, or you're taking a FSID as an argument, should be roughly the same problem right?

(e.g. it's a more generalized problem, will need a more generalized solution)

[severe atlas]

We already mount all the arguments that are FSIDs into the container, the fix here was basically to do the same thing but also for the parent result now

But yeah, we need a better solution for all this

It needs to be more configurable at the very minimum, while still retaining the caching behavior in all cases

[minor furnace]

Yeah, which was already a workaround-ish ...

maybe it's tied to the FSID to ID problem (that prevented hashing the FS)? Tied as in, same solution

if bk would give us the CONTENT hash for the def

then we would use that

both problems solved, elegantly

(and also convert content hash back to def)

[minor furnace]

Awesome, thank you!

[severe atlas]

I don't know if that solves the FSID to ID problem. I was thinking about that more and realized that even if we add the ability to request results based on hashes (whether content based or not), it doesn't guarantee that buildkit's cache actually still has the result. It gets extremely subtle, involves what buildkit can and cannot prune based on the layering of cache content, etc. But I think that problem is going to be really tricky actually.

[minor furnace]

right

[severe atlas]

Last night I tried out using zlib to compress the FSIDs. It feels incredibly silly but does make a huge difference, like 6-7x smaller for some basic alpine examples.

I was also thinking about how we could truncate the FSIDs when printed for informational-only output. Maybe I'm missing something or severely overthinking it, but it's hard because A) we can't just truncate them in every case (e.g. override their serialization methods) because sometimes we do actually need them to be serialized to the full value B) they get printed as part of all sorts of random calls (e.g. errors generated by the graphql client libs) and C) we need to implement this per-sdk.

[minor furnace]

and D) if you want to copy-paste the ID to debug the error, you're going to have a bad time

D could drive a sane person mad

[severe atlas]

Yeah true. The only other thought I came up with was that you can return hashed FSIDs to clients, but when a runtime creates /output/dagger.json it makes API calls back to cloak to get the full LLB result and writes those to dagger.json. I think this would result in cloak always knowing the mapping but clients not always being exposed to the full b64 blob. But I really, really hope we can think of something better than that because it's a huge annoyance for SDK runtime authors to deal with.

And also, it would have been nice if instead of the runtime doing that, cloak could do it as a layer on top of the /output/dagger.json result, but that would mean it's possible for that top layer with the full LLB to be pruned even if the lower layer with the hashes stays, so not a viable path (besides the fact that it's still very ugly).

My only final vague thought about a solution is that if we fully wrap buildkitd, we can do a lot more in terms of having direct control over and access to buildkit's state, containerd garbage collection, etc. which may allow us to solve this without having to do something dumb like create our own separate boltdb for storing the mappings. However, we'd probably be playing with fire there, need to see what Tonis thinks in terms of which interfaces have any guarantee of stability (if any).

[minor furnace]

@severe atlas is it still extensions or sources now?

[severe atlas]

It's extensions at the moment, I'm not changing anything there until after the demo is done

[minor furnace]

hmm

  18 + │ extensions:
  19 + │   - path: extensions/deployxx
  20 + │     sdk: ts

(deployxx doesn't exist, on purpose)

looks like it's ignored

(so i tried to put something not existing, and no failure)

[severe atlas]

What command are you running that results in nothing happening? generate?

[minor furnace]

dev

[severe atlas]

Huh I just tried this:

name: examples
extensions:
  - path: fake/notreal
    sdk: ts

Ran cloak dev and it failed. It complained about build not existing (the ts runtime tried to build the non-existent package.json). If I switch to the go sdk it's a more straightforward stat /src/fake/notreal: directory not found error

[minor furnace]

@severe atlas [random] I was walking through the code, noticed the ~/.ssh/known_hosts hack. There's a better workaround with IIRC GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=no"

that should work for any git repo, not just github

[severe atlas]

Right but I don't know if I want to disable that since it's a security feature. Buildkit has some builtin feature to actually lookup the key for you and put it in your container, I just didn't have time to figure out how to use it yet, gonna try that as a fix first

[minor furnace]

uhm

still not failing for me, fresh build from your branch. Added to todoapp/cloak.yaml

[minor furnace]

Yeah, but unless we parse the user's go.mod / package.json, we won't know what to put in there

[minor furnace]

latest todoapp, latest workflow-clean branch

diff --git a/cloak.yaml b/cloak.yaml
index 265e4b6..aa692af 100644
--- a/cloak.yaml
+++ b/cloak.yaml
@@ -15,3 +15,6 @@ dependencies:
       remote: [e-mail]:dagger/cloak.git
       ref: demov2
       path: examples/netlify/ts/cloak.yaml
+extensions:
+  - path: fake/notreal
+    sdk: ts

[severe atlas]

I was just going to put github's in there for our runtime LLB since that covers an enormous number of cases, can generalize it later by making it configurable, etc.

[severe atlas]

trying now

[severe atlas]

Yep in that case it doesn't fail for me either... looking into it

[minor furnace]

•           if err != nil && !isGatewayFileNotFound(err) {
  

•           if err != nil {
  

I think there's a silent error

Error: failed to solve: input:5: core.filesystem.loadProject open /tmp/buildkit-mount2598725652/fake/notreal/schema.graphql: no such file or directory

[severe atlas]

That's leftover from a previous iteration where schemas were optional in extensions, agree we should remove those checks but there's a more fundamental problem with the ts runtime (it currently fails if you switch to sdk: go). What's happening is that the runtime tries to build the code by setting the LLB dir to fake/notreal , which results in buildkit automatically making the directory when it doesn't exist. Then it runs yarn install and yarn build from there, which results in yarn going up a few directories where it finds the package.json of todoapp itself and then runs install+build there 😑

[minor furnace]

ah

[severe atlas]

Just gonna not use llb.Dir and instead use --cwd w/ the yarn calls

[minor furnace]

yeah

oh

so I did the non-existing-test because my extension was not loading

but what's the situation with yarn

do i have to put a package.json in the extension?

[maiden orbit]

@minor furnace @severe atlas I'm joining a rehearsal call now but, any progress on that step 4 - todoapp extension? Would be really perfect to have

[minor furnace]

I almost have it, stuck on a few runtime bugs

[maiden orbit]

Awesome thanks!

[severe atlas]

I think you do at the moment yeah. It should be possible to re-use todoapp's too, but right now because it's hardcoded to call yarn build it's awkward. If we add the ability to provide args to the runtime and you could override the target then you could re-use todoapps

[minor furnace]

👍

I think that's why it was silently failing, yarn going up to the parent

weird it was silent, no time to dive into it

[severe atlas]

yes that's what was happening

[minor furnace]

(we are making many assumptions in the ts sdk, for instance dist/index.js etc, there should be a way to work better out of the box, and provide options for what's missing)

also in theory it should be "node" sdk, i don't think it matters if it's TS or JS

(code generation should always do TS + .d. files so it works on both)

err

JS + .d. files

[severe atlas]

Yep this was just a 1-to-1 translation of the dockerfiles for now, just to get rid of that boilerplate. Once we migrate these to be actual extensions themselves rather than hardcoded LLB that will be the time to start cleaning+generalizing them up

[minor furnace]

(just dumping my todolist as i see stuff)

[severe atlas]

yarn --cwd fake/notreal also doesn't result in an error (???), confirmed testing manually on my host, so I guess I'm going to just do a cd since that will actually fail...

Pushed commit w/ the fix so it fails now

[minor furnace]

@severe atlas almost there ...

now [cloak generate] doesn't generate the new stuff. investigating, surely a problem on my end, but it's a silent error again

[severe atlas]

If you are using the TS sdk there isn't much that gets generated, we only export the schema and operation files to the gen dir for now

Are you getting those at least? or just nothing?

[minor furnace]

also, note for another day: slightly annoyed at the "one workflow per directory, workflow must has a main.go file" constraint

[minor furnace]

no it's TS to Go

there should be stuff

I added an extension, ran cloak generated, expected magefiles/gen to contain it

@severe atlas huh, weird. if i rm -rf magefiles/gen, cloak generate, there's nothing

[severe atlas]

Can you send the cloak.yaml (or push to separate branch), I'll try to reproduce it then

[minor furnace]

@severe atlas https://github.com/dagger/todoapp/tree/cloak-demo-v2-step-4

[severe atlas]

When I run cloak generate on that branch nothing happens, but that's because everything has already been generated I think. If I delete extensions/deploy/gen and re-run cloak generate, then the gen dir gets recreated. So that seems expected so far.

But when I delete api/magefiles/gen and then run cloak generate from the root of the todoapp repo it actually fails, which is technically expected I suppose because the go.mod that it needs access to is in api/magefiles. So that's different than your silent failure but still weird and probably deserves a fix. If you run the commands in the right order in the demo it would be fine, but that's kind of treacherous

[minor furnace]

yeah first part works, second part though what's the right order?

~/w/todoapp ❯❯❯ rm -rf api/magefiles/gen                                                                                        ✘ 130 cloak-demo-v2-step-4 ✭ ✖ ◼
~/w/todoapp ❯❯❯ cloak generate                                                                                                        cloak-demo-v2-step-4 ✭ ✖ ◼
#1 local://.workdir
#1 transferring .workdir: 78.51kB 0.0s done
#1 DONE 0.0s

#2 copy / /
#2 DONE 0.0s

#3 git://github.com/dagger/cloak.git#demov2
#3 1.400 ca9da904b4dd02e226dd8b42f0ca0350ace49584    refs/tags/demov2
#3 CACHED
/Users/al/work/todoapp/api/magefiles/main.go already exists, skipping generation
go: finding module for package github.com/dagger/todoapp/api/magefiles/gen/yarn
go: finding module for package github.com/dagger/todoapp/api/magefiles/gen/core
go: finding module for package github.com/dagger/todoapp/api/magefiles/gen/netlify
github.com/dagger/todoapp/api/magefiles imports
    github.com/dagger/todoapp/api/magefiles/gen/core: module github.com/dagger/todoapp@latest found (v0.0.0-20220818170310-4f8733ac1b2f), but does not contain package github.com/dagger/todoapp/api/magefiles/gen/core

(basically complains because they were never re-created)

it's not "silent" but the generation fails

[severe atlas]

Yeah that's because you already have code in the dir which has those imports. We have to run commands that interact with go.mod (actually gqlgen and genqlient do too surprisingly), and when we do that the go command decides it needs to find the imports you declared and complains cause it can't find them...

The order that I imagine we would follow in the demo and that works is that you start with no go code, then run cloak generate, which creates the gen dir and then add your go code that references it.

[minor furnace]

i see

well that was another test

my end-goal problem is this:

~/w/todoapp ❯❯❯ cloak generate                                                                                                          cloak-demo-v2-step-4 ✭ ◼
#1 local://.workdir
#1 transferring .workdir: 116.69kB 0.1s done
#1 DONE 0.1s

#2 copy / /
#2 DONE 0.1s

#3 git://github.com/dagger/cloak.git#demov2
#3 1.602 ca9da904b4dd02e226dd8b42f0ca0350ace49584    refs/tags/demov2
#3 CACHED
/Users/al/work/todoapp/api/magefiles/main.go already exists, skipping generation
~/w/todoapp ❯❯❯ ls -l api/magefiles/gen                                                                                                 cloak-demo-v2-step-4 ✭ ◼
total 0
drwxr-xr-x  6 al  staff  192 Aug 30 18:38 core
drwxr-xr-x  6 al  staff  192 Aug 30 18:38 netlify
drwxr-xr-x  6 al  staff  192 Aug 30 18:38 yarn

the new extension is not being generated

I see it in the dev api server

and i successfully call it from TS

(yarn deploy-3)

but it just doesn't get generated

no error messages, nothing

Ohh could it be because of the extend filesystem?

[severe atlas]

I ran into that go error yesterday but now thinking about it again I could probably try to run the client codegen first, which may allow me to skip some of the go commands that cause the current failure. However, that will rely on genqlient also not running go mod commands that break, which may or may not be true.

[severe atlas]

What are you expecting in terms of the generation? When I ran cloak generate I got extensions/deploy/gen created, which is all that's expected at the moment

[minor furnace]

I've loaded an extra extension

extensions/deploy

there's operations in there

I'd like to use them in the magefile

[minor furnace]

however it only generates core/netlify/yarn, not my extension

[severe atlas]

Ohhh okay sure, yeah that's actually not handled yet. It will only generate client stubs for what you have declared dependencies on

[minor furnace]

ohh

[severe atlas]

It makes perfect sense though

We used to have the behavior of generating client stubs for your own self a while back, got rid of that because it was weird

I guess now it makes sense to add it back in a slightly different form

[minor furnace]

I'm doing the demo step 4 here basically

One notch after: create an extension with the common stuff (todoapp deploy), share it across TS and Go

I'm using it in TS, I was hoping I could use the stubs from the magefile to invoke it

[severe atlas]

Yep that's the new frontier today 🙂 I can probably make a quick change so that workflows get client stubs for extensions in the project in addition to dependencies

let me try quick

[minor furnace]

Thanks 🙂 if it's quick that'd be amazing

as a fallback I'll make a manual gql query

but codegen would be amazing

@severe atlas [for later]: https://github.com/dagger/cloak/pull/95/files/0d072607b2b02f1799ea978ca1090c4e96c25fb0..c7cd4fd7d1073039b09337538dbbe97ce97bbbd5#diff-9c4d8ebe166956ed36cc2c557f3ec3bbe1dc3392a429f4c7f49e6d138e1f0e1fL94

this can probably be removed as well

oh, actually not

[severe atlas]

Pushed the change, once pulled you can run cloak generate and looks like the clients are created

I also pushed a commit to your branch in todoapp, you had a typo in the operations.graphql

[minor furnace]

yay! thank you!

@severe atlas okay running into another problem, maybe you already solved it:

the extension is extending Filesystem, adding todoAppDeploy()

The generate return response though doesn't contain that field

DeployResponse -> DeployCore -> dagger.Filesystem -> (and here we lose the todoAppDeploy pointer)

[severe atlas]

Oh I see, no I didn't run into that when I added the filesystem extension because I was just doing everything w/ TS. I guess the client codegen needs updating to handle that better

[minor furnace]

yeah

I think the solution would be not to generate by extension, but generate in bulk

there's interdependencies between different extensions

[severe atlas]

I can try, but you should also consider not extending Filesystem there. When I go to battle w/ the codegen frameworks it usually takes hours to convince it to do what we want (which is why we need to switch away to our own more customizable framework in the future)

[minor furnace]

yeah i'm doing that right now

no time for the alternative 🙂

[severe atlas]

The problem is just that we are associating an existing pre-made model with Filesystem and that one doesn't know about the new extension. I will try a very small change on the offset it works, but yeah not optimistic

[minor furnace]

but taking a step back, I think generating stubs for the "whole API" rather than bits by bits makes more sense. The schema is intertwined between exteions etc

after all that's how it would work for a "regular API"

I think it's this way because of legacy reasons, unless I'm forgetting something

It would probably make our lives easier in so many ways -- like the loadedschema/executableschema/etc no need for that anymore, we could literally generate the API from an out of the box introspection query

anyway

converting to non-fs

[severe atlas]

There are advantages to having generated clients in their own packages for each dependency in terms of namespace conflicts. I also don't know if it invalidates the need to separately load and compile schemas, but yeah discussion for another day

[minor furnace]

only downside is, it will be MORE verbose using the extension than without, in TS

because in deploy-2 we were chaining fs + yarn + deploy, here i'll have to do one query for FS and then one for deploy

[severe atlas]

I did manage to get the generation to work as expected here w/ a one line change, but I would need to test more to make sure it didn't break everything else. So plausible if you think it makes a big difference, but probably not worth the risk if it is only a small thing

[severe atlas]

I think I'm fine with that, it's too risky to change stuff unnecessarily atm. I doubt anyone will really notice, can fix it for the next demos

@maiden orbit is 12pm when we are running through the demo again? Or before? If we are adding the step 4 w/ extensions I'll need to retag demov2 with the fix mentioned above (need to generate clients for extensions in the project in addition to just dependencies).

[maiden orbit]

@minor furnace rename summary:

scripts/deploy-1 becomes scripts/kick-the-tires
scripts/deploy-2 becomes scripts/deploy

[severe atlas]

Realized there's a very simple way to improve this, I just reordered it so client stubs get created first, then the rest of the codegen runs. Now it works when you pull your branch, delete api/magefiles/gen and re-run codegen

Will make the demo safer in terms of not having to always follow quite as stringent an order of operations, so pushed it: https://github.com/dagger/cloak/pull/95/commits/eb9c66ad0a35a0b1aca68409675873070bf7bb35

[minor furnace]

ugh

sorry

i just push -f'ed into the branch

:sadpanda:

[severe atlas]

Oh my push was to my workflow-clean branch, no changes needed on the todoapp one

[minor furnace]

oh perfect

need your help on one last bit

yarn deploy-3 -> works alright

mage deployExt -> caching issue, like before

no idea what's going up

[severe atlas]

Immediate guess is that you are using an older version of the engine

It's inconsistent at the moment between go and ts

TS engine shells out to cloak binary

Go engine is a library

so make sure your go.mod is updated to the latest commit

[minor furnace]

oh right

[severe atlas]

That's another FIXME, but for later

Should I retag demov2 with my latest fixes then?

[minor furnace]

should i grab demov2 or your branch?

[severe atlas]

Haha, well okay, I will retag it in that case

[minor furnace]

yep

can you make it a branch while you're at it?

can't git pull the tag

[severe atlas]

Sure

[minor furnace]

we'll tag it after the demo

[severe atlas]

It's named demov2-branch because I don't know what happens when a tag and branch have same name: https://github.com/dagger/cloak/tree/demov2-branch

Ah but wait, there's a stupid hack in cloak generate where we are hardcoding the dependency on the branch (so there's less to type during the demo...) https://github.com/sipsma/cloak/blob/eb9c66ad0a35a0b1aca68409675873070bf7bb35/cmd/cloak/gogenerate.go#L39-L39

[minor furnace]

ok running this: go get -u github.com/dagger/cloak@demov2-branch

[severe atlas]

Sure that's fine for now, just don't run cloak generate until I push another commit

[minor furnace]

yep

(btw the ssh agent thing is not working well on my end :|)

i'm using secretive so my ssh keys are on my macbook's secure enclave. works fine with ssh and git, but not with cloak (had to ssh-add manually)

my guess is we're not exporting some bits

doesn't matter

[severe atlas]

that's what I'm using too, it worked for me

[minor furnace]

oh

weird

[severe atlas]

but who knows, I setup that up a long time ago so I have no idea what configuration I did anymore

[minor furnace]

I had to use my "the laptop fell in the bath tub ssh key" 🙂

[severe atlas]

Haha I mean that works too for now

[minor furnace]

actually, go get isn't working out for me

oh

what's the go proxy thing?

[severe atlas]

You can now also run cloak generate, pushed the fix for setting it to the branch name

[severe atlas]

Oh did you setup ~/.gitconfig?

[minor furnace]

not correctly apparently 🙂

but working now

[severe atlas]

[url "ssh://[e-mail]/dagger/cloak"]
  insteadOf = https://github.com/dagger/cloak

[minor furnace]

i've set GOPRIVATE manually

[severe atlas]

Ah that too, for cloak generate I set that automatically, I forgot you are doing this manually rn

[minor furnace]

argh

actually forget the branch

there's also cloak.yaml

too many changes

[severe atlas]

Haha okay, one sec, retagging

[minor furnace]

let's deal with the dag one last day. wdyt?

*tag

sorry about the extra useless steps 🙂

99% there, just verifying everything, that the cache works etc

[severe atlas]

No worries at all, pushed the tag, just be sure to do this to be safe #1012827742481428560 message

[minor furnace]

Yeah, especially buildkit, I guess it puts git tags in cache forever

if it doesn't it definitely should

[severe atlas]

Both yarn and go running on the host ran into errors too under certain conditions (refused to pull when it would overwrite a tag they had pulled before). So yeah just blasting everything is safer now

[minor furnace]

  type: 'system',
  errno: 'ECONNREFUSED',
  code: 'ECONNREFUSED'

still getting this with a clean cache, first time

[severe atlas]

That's weird, I don't think I've ever gotten that

is macos your host system? I'm running everything inside a linux vm. Don't know if the engine lib from the sdk could somehow have subtly different behavior. The fact that it happens with a clean cache makes me suspect some timing bug, but idk

[minor furnace]

ok, it works!

cache is not broken anymore

thanks!

[severe atlas]

Actually, if you have sec and care enough to debug this further, run cat node_modules/\@dagger.io/dagger/dist/engine.js from the root of your todoapp, I want to make sure you have the latest

[minor furnace]

i'll finish this stuff first

debug tomorrow 🙂

[minor furnace]

ok, everything works

[minor furnace]

taking care of this

then creating the extension branch

will send a recap in ~5 min

TextHTTPError: Unauthorized
    at parseResponse (file:///Users/al/work/todoapp/node_modules/netlify/src/methods/response.js:13:35)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async callMethod (file:///Users/al/work/todoapp/node_modules/netlify/src/methods/index.js:32:26)
    at async file:///Users/al/work/todoapp/scripts/kick-the-tires/index.mjs:122:14
    at async Engine.run (file:///Users/al/work/todoapp/node_modules/@dagger.io/dagger/dist/engine.js:47:9) {
  status: 401,
  data: ''

@severe atlas another one 😐

[severe atlas]

That's netlify saying you have the wrong auth token I think

[minor furnace]

ah, no that's just me being stupid

different terminal

getting tired 🙂

[severe atlas]

No worries, me too 🙂

[minor furnace]

@maiden orbit @severe atlas @viscid tartan

Ok everything is ready:

1. todoapp cloak-demo-v2

• demo as usual (yarn kick-the-tires and yarn deploy)
• contains extensions/deploy but it is NOT used, not referenced

2. todoapp cloak-demo-v2-extension

• extension based demo
• changes cloak.yaml to load the demo
• changes scripts/deploy and api/magefiles to use the extension

⚠️⚠️⚠️ You need the latest demov2 tag from cloak. Need to build the binary from here. Also need to wipe the buildkit cache, go cache, yarn cache. See pinned message by @severe atlas