Otherwise there will always be a

[hallow crest]

It removes it from what I suspect is the most common case where you are running these automation steps from the root of your existing repo (like todoapp). E.g. I have my package.json at the root of my repo, I just put my cloak.yaml next to it and then run my automation steps. I don't need to specify CLOAK_CONFIG because the default is to find it in the current working dir.

It doesn't handle every case though, you'll still need to specify it if you are running outside the root of your project dir. I think it just works better because it matches the goal of workflows, which is to integrate with existing toolchains, and many (or most?) of those toolchains have the same concept of a "project root" which contains configuration.

I could easily be wrong in my guesses here though.

[modern cave]

You're right that it's definitely more common to run these from the project root. Will remove CLOAK_CONFIG= in at least those common cases

[hallow crest]

I agree that I'd love to remove the need to read the file at runtime (though maybe it should always be possible to override).

I just don't think embedding the file w/ the code works unless we accept extra boilerplate for the user.

There are probably other possibilities, like what I mentioned a while back about ways we could eliminate the need for yaml entirely, but they are fairly complex if possible at all.

[modern cave]

I have a strong feeling though that project-centric won't solve it for everyone. There's bound to be complicated repo layouts, a combination of tools that handle workdir in an unusual way, etc

maybe monorepos

It may end up being a tradeoff of "more complexity for the end user" vs "more complexity for the workflow dev"

[hallow crest]

I completely agree, there will still be non-obscure cases that require setting CLOAK_CONFIG, I just am trying to optimize for my guess at the most common case.

[modern cave]

If we made laoding extensions in graphql the norm, this problem would go away entirely...

Generated stubs for each extension could load their respective extension when initialized

[waxen isle]

I think defaulting to the root is the best option, in the JS ecosystem, all config files are generally in the root (Next.js, Vercel, Netlify) and if necessary, the library has a .[libraryName] folder

Here's an image from our /website folder in dagger/dagger

And introducing a flag to point to another location in case the config file is not there should work IMO

[hallow crest]

I was just pondering this and then also remembered something Andrea and you brought up before about wrappers, which lead to this:

You invoke workflows with e.g.

cloak run -p ./workflows/build node index.mjs

or

cloak run -p ./workflows/deploy go run .

Can debate the command name run, but point is that you still provide the actual command to be exec'd (i.e. node ...) but with this cloak run wrapper. We already need cloak to exist in order for cloak generate, so not a new requirement for a binary.

The point of the wrapper is to do any "environmental" setup needed to actually execute the workflow while avoiding the ugliness of env vars. So for the examples above cloak run would handle starting up the engine (nice because it simplifies the sdk), changing directory to dir containing cloak.yaml and then fork/exec'ing the rest of the command provided.

Pros:

• Less ugly than CLOAK_CONFIG (by using a cmdline flag)
• Simplifies SDKs (don't need to have them find cloak binary and start engine, they just assume it's running and connect)
• Can switch between workflow-centric and project-centric pretty seamlessly if needed I think; this makes our current demo case less ugly but also doesn't invalidate a future switch to project-centric if desired.

Cons:

• A new subcommand that needs to be named well, documented, understood by users, etc.
• cloak run ... go run ... looks weird?

@modern cave WDYT? Sorry I'm going way off a tangent from what you were saying, it just seems promising and simple enough an idea.

Obviously we are running quite low on time. I'm about 50% done w/ my switch to project centric as described before. If you think this new suggestion is absolutely better then I'll switch to trying it out. Will continue w/ project centric impl until I hear back

Also, cloak dev would probably go away w/ this. If you do cloak run -p ... with no subsequent args (i.e. what to fork exec) then you can just end up with what cloak dev does today

[modern cave]

@hallow crest let me think for a second

[hallow crest]

For sure, I have project centric working now either way, so we have that as an option too.

[modern cave]

Reminds me of the env command

cloak env ?

[hallow crest]

@raven lodge brought this up as an example of wrappers previously, same idea: https://github.com/99designs/aws-vault

cloak env ?
Sure, I like that. It sets up the env for executing your command

[modern cave]

cloak env go run ./workflows/build/foo would literally be equivalent to env CLOAK_CONFIG=./workflows/build/foo/cloak.yaml CLOAK_HOST=localhost:8080 go run ./workflows/build/foo

I guess it would be more than that if it actually bootstraps the engine

It's a big decision though. Drastically reduces what happens inside the lib

Also guarantees that standalone execution will never work

[hallow crest]

True, or at least it means that standalone execution is a different concept. There could still be a library that works the same way as the current demo v2 (spins up cloak). But then we have to manage both ways of executing workflows, users have to be aware of both, etc.

[hallow crest]

lib is just a client in this case essentially. can read which endpoint to connect to cloak over from an env var set by cloak env

[modern cave]

Yes and once we define these env variables (eg. CLOAK_HOST) we have to document its meaning and not break its behavior. So it locks us into a particular model too

That particular one scares me because of host access. We don't want people starting a remote engine and connecting to it over the network by hijacking CLOAK_HOST. We need the first engine you talk to, to always be local, for host { } to work as intended

That's one reason I like the current model of each SDK being tightly coupled to engine bootstrap. In fact I think we should take it further and eg. connect to the engine through a named pipe or temporary unix socket, instead of a tcp port (I'm guessing you didn't plan on keeping that tcp port anyway)

Honestly project-centric scares me less at this particular moment

• relying on workdir the way yarn does

in other words: "your cloak engine needs to run in a cloak project directory. Make sure your workdir is inside a cloak project. The end."

We can always stack the wrapper idea on top later

"your cloak engine needs to run in a cloak project directory. Make sure your workdir is inside a cloak project.

... as a convenience you can set the workdir with cloak env/run/whatever --workdir
"

[hallow crest]

Yeah it should just use an anonymous pipe passed during fork actually, tcp is just cause I could re-use cloak dev for now.

Point well taken about CLOAK_HOST...

in other words: "your cloak engine needs to run in a cloak project directory. Make sure your workdir is inside a cloak project. The end."
We can always stack the wrapper idea on top later
SGTM, I am about to push the changes to the todoapp repo for project centric updates so you can see what that looks like

@modern cave https://github.com/dagger/todoapp/commit/92c49612ad765e17ac51adb2675580af298f1f3d

The generate command is very slightly different than what I posted this morning w/ the project changes: cloak generate --output-dir workflows/deploy --workflow

[modern cave]

Nice thanks.

[hallow crest]

all the build+deploy scripts in package.json are working for me locally though

[modern cave]

My first thought was: "do we need the sources field?" But I guess we do for SDK selection, and later for sdk-specific settings

And possibly workflow-specific dependencies if we decide we want them (project-wide is fine right now)

[hallow crest]

Oh yeah, also right now sources is also for both workflows and extensions. The only thing that makes an extension different is that it has a schema.graphql file (and you run cloak generate --extension).

Very easy to switch though if having separate extension and workflow keys is preferable

Just made sense to me at the moment and saved a few minutes

[modern cave]

Let me think about that one..

yeah I think separate extensions and workflows would be simpler to show and explain

Also leaves open the possibility that the same directory could be used as one or the either, or both? 🙂

[hallow crest]

Ah sure, that makes sense, okay I'll make the update but hold off on pushing until you're ready so I don't break todoapp suddenly

[modern cave]

You can go ahead and push anytime, I'll just wait for the right time to use the new cloak from your branch

[raven lodge]

Hey, catching up on the last few days of changes. Did we switch to project-centric? Is this final or a test?

[hallow crest]

Yep switched to project centric. It's not finalized by any means. I personally think it may be on the right track with a few other modifications, but totally open to anything still.

Also, it's still just in my PR. I am working on getting everything there merged into main now, but will take a while

[modern cave]

Yeah it’s whatever helps with the demo for now. Then we can assess

[raven lodge]

Raw feedback, without context: I find the number of concepts a bit overwhelming, Even after reading the code I don't fully get it (workflows have schemas now? what's the difference with extensions? and projects also have schemas?). Still reading through to understand, will need some guidance over chat, but I find it "harder" than before (used to be pretty "dumb" and straightforward)

Also a bit concerned about internal complexity... the scope of internal packages is less defined, they're getting mixed together. The exposed surface area grew quite a bit (e.g. engine now exposes a Project struct from the Core API which used to be private, which itself now exposes fields from the internal extension package, which used to be private). There's a lot of nested for looping going on to handle projects, we'll need to rethink about parallelism, error handling, testing, etc

But again -- I literally have zero context, I'm sure there's very good reasons, looking forward for a catch up

[hallow crest]

Ah yeah a lot of those are just artifacts of the fact that I was making these changes very quickly and only updating what needed to be update to get the demo functioning. That's why I didn't merge into main yet. For instance, I didn't even realize that I left schema and operations in the workflow schema, that's just a typo 🙂

[raven lodge]

Oh, ok! Makes more sense now 🙂

Anything in particular I should be looking at tomorrow?

Otherwise my plan is to catch up on the latest changes, help out Gerhard with his demo, and a EU community call

[hallow crest]

My plan is to separate out the commits that are "low-controversy", merge those in first, then I'll clean up the code for the parts that need more thought+discussion (like the project related stuff). I agree right now it makes no sense looking at the code without prior context, I didn't update most of the internal names of structs and stuff yet. Once that's updated I'll let you know.

Anything in particular I should be looking at tomorrow?
I think I got most of the stuff needed for the demo in, so nothing high priority

But if possible would be good to sync up tomorrow morning, maybe after the community call?

[raven lodge]

Sounds good. There were so many discord messages that I thought the quickest way to catch up was to read the PR

[hallow crest]

Yeah that would normally make sense, things just were moving a million miles an hour last week... but now there's time to catch up and clean up

[raven lodge]

Yep, I'll be available up until at least 8pm (your 11am)