How to familiarize myself with Cloak?

[fast finch]

Re-posting a thread started by @daring pike :

Hey all :ferrisWave:

Is there somewhere I could familiarize myself with the current state of the Cloak project?

I'm working on overhauling our CI setup at work in Q4 and was looking to use Dagger. Based on what I've gleaned it sounds like Cloak is the way forward, so it would be nice to build on top of it from the start.

I might be able to make [the community call]. If not, I can always catch the recording.

I have been lurking in the cloak-dev channel, there's just a lot to take in all at once 😄

In terms of what we're looking to get out of this CI overhaul:

• Right now our CI pipeline is largely YAML soup. Would love to get this cleaned up and make it more maintainable (e.g., sharing modules for common CI tasks)
• We have a monorepo setup, so trying to figure out a good way to do automated change detection (only running CI blocks based on which projects have been changed and the projects that depend on them)
• Faster CI. In particular, we spend a lot of time rebuilding the same artifacts over and over. Even with a cache in place, it can take a while to hydrate the build artifacts from the cache. Looking at we might use something like Docker to speed this up
• Not totally sure if we want to do it this quarter, but at some point we want to switch CI vendors, so using Dagger to abstract away the details of our current CI vendor would make this switch easier when we're ready

[fast finch]

Hi @daring pike , were you able to watch the community call recording? I'll try to answer a few of your questions here, at least partially

[fast finch]

Faster CI. In particular, we spend a lot of time rebuilding the same artifacts over and over. Even with a cache in place, it can take a while to hydrate the build artifacts from the cache. Looking at we might use something like Docker to speed this up

Dagger can definitely help with this. It uses buildkit under the hood, so all the caching tech available to Docker is available to Dagger as well. The biggest benefit compared to using each CI's proprietary artifact caching, is that caching works out of the box anywhere, including on your local dev environment.

The tricky part is usually persisting the Dagger Engine's cache when it runs in ephemeral worker machines - which is usually the case in CI. Buildkit supports a variety of backends for this: docker registry, S3, etc. In Dagger 0.2 you can control these settings. See https://docs.dagger.io/1237/persistent-cache-with-dagger

In cloak (aka future Dagger 0.3), these configuration settings are not yet available, but we plan to. It's just waiting on a few core design decisions. @muted fog or @winter patrol (plus @@sipsma who is on vacation) can answer more technical follow-up questions on this point

Not totally sure if we want to do it this quarter, but at some point we want to switch CI vendors, so using Dagger to abstract away the details of our current CI vendor would make this switch easier when we're ready

Indeed 🙂

We have a monorepo setup, so trying to figure out a good way to do automated change detection (only running CI blocks based on which projects have been changed and the projects that depend on them)

There is a short term answer and a longer-term answer.

• Short term: use your CI provider's proprietary change detection feature, if they have one. Then hand off to the corresponding Dagger pipeline (implemented in the language of your choice). Don't worry too much about calling the same pipeline multiple times, or other edge cases: all pipeline invocations will share the same cache, so any duplicate invocations will just return instantly.

• Longer term: disable conditional execution in your CI - removing one more dependency to a proprietary feature. This means every Dagger pipeline on the monorepo will be called every time. But that's OK because 1) they all share the same cache, so pipelines with unmodified inputs will just do nothing, and 2) cache lookup overhead is so small, that the simplicity of handing off the problem to Dagger comes at no performance cost.
  The reason this is a longer term answer, is that point 2) is not yet true: in a CI environment with a typical caching persistence backend (registry, S3, Github storage..) the overhead of running a fully cached pipeline is still large enough that it's worth filtering them in CI first if you can.

cc @fast lodge (on vacation today) who has dealt with this operationally for our own CI

[daring pike]

Hey @fast finch, I was actually able to join last week’s community call 🙂

I found it very useful in getting a better understanding of the current state of things.

[fast finch]

@solemn prism 👆🏻

[daring pike]

One follow-up question I have is what's the best way to try out Cloak (both locally and in CI) if I want to build out a POC?

Are there pre-built binaries for Cloak available (or coming soon)?

[fast finch]

cloak will be primarily consumed as language-specific SDKs. CLI too but in a few weeks (want to polish the shell scripting experience first)

Go SDK is just about done, launching next week

[daring pike]

Oh, I see, so instead of invoking it like dagger do build like with Dagger v0.2 there would be some different entrypoint (in TypeScript, for instance, ts-node run-ci.ts) and that would pull in the Cloak TypeScript SDK and invoke it that way?

[fast finch]

exactly. Your code is the entrypoint

rough diagram here: https://devel.docs.dagger.io/fssrz/sdk-go/#how-it-works

[daring pike]

Nice!

I can see here in the todoapp example that it's importing from @dagger.io/dagger: https://github.com/dagger/todoapp/blob/72719d085623267d2cd6fb196ddebc3ebe9ec956/scripts/kick-the-tires/index.mjs#L2

But I'm not seeing the package on npm; I'm assuming it may be private.

I was talking with @abstract pasture about the TypeScript SDK a bit at the Dagger community call two weeks ago, and he mentioned that it was functional, just missing the codegen aspect (so you have to write the GraphQL by hand).