Buildkit resources

[royal wigeon]

[2017] Buildkit introduction blog post. https://blog.mobyproject.org/introducing-buildkit-17e056cc5317

Turning on the CrazyMax signal 🙂 @plush compass

Project README of course: https://github.com/moby/buildkit/blob/master/README.md

There is an architecture diagram somewhere, but I can never find it

There’s an examples/ directory showing how to assemble and invoke Buildkit DAGs directly from the Go API. It’s pretty unreadable TBH, a good reminder of why Cue+Buildkit is such a superior combination. https://github.com/moby/buildkit/blob/master/examples/gobuild/main.go

[limpid sequoia]

https://giphy.com/gifs/nbc-the-blacklist-42HJ26S7bvbmVhcaR2

that's me. just less hair.

[royal wigeon]

https://tenor.com/view/hole-rabbit-hole-alice-in-wonderland-disney-animated-gif-5928662

[royal wigeon]

@faint cedar not sure if you still watch this channel… But if you do we would welcome any suggestions 🙂

[limpid sequoia]

The build cache can even be exported to a registry, where it can be pulled on-demand by subsequent invocations on any host.
🤯
have we explored that yet?

[royal wigeon]

Yes 🙂 It has drawbacks, but current plan is to rely on that for CI embedding

They have a cache exporter that works specifically with Github Action’s persistent storage API

cc @lapis yarrow @plush compass

[royal wigeon]

(Main drawbacks are that exporting is an additional action in the DAG, and can’t be parallelized, so it adds to the total run time. And some remote cache exporters don’t support “max mode”, meaning they don’t actually cache everything. This can result in only caching results from the last run, greatly reducing utility of the cache.)

[plush compass]

This dev guide can be useful too. Will see if we can port (and update it) to the repo in /docs.

[timid gust]

I was gonna try to find some resources around Buildkit. Thanks for pulling these together!

[royal wigeon]

LLB protobuf definitions if anyone else is curious 🙂 https://github.com/moby/buildkit/blob/master/solver/pb/ops.proto

A thought just occured to me: does Cue support native Protobuf import? 😄 cc @timid gust

[timid gust]

@royal wigeon Hmm.... I think so. Can't remember if that work is done, or just in progress.

[timid gust]

https://cuelang.org/docs/usecases/generate/#extract-data-definition-from-existing-sources

CUE can currently extract definitions from:

Go code
Protobuf definitions.

[royal wigeon]

Nice! So could I for example cue get github.com/moby/buildkit/solver/pb ?

Or cue import ?

[timid gust]

Looks like it's through the API for now: https://cuelang.org/docs/integrations/protobuf/#extract-cue-from-a-standalone-proto-file

[royal wigeon]

Yeah it doesn’t really say how

[timid gust]

@royal wigeon

[royal wigeon]

Ah cue import does mention proto support! Will try that

[timid gust]

Nice!

[royal wigeon]

@timid gust it seems to be OK with the proto file, but failing on import (the proto file has dependencies on other proto files)

Dropping for now, I don’t really need it for my proof of concept. Copy-pasting bits and pieces is fine for my immediate purpose.

[timid gust]

@royal wigeon 👍 -I, --proto_path stringArray paths in which to search for imports

[royal wigeon]

TIL, buildkit LLB does not define distinct operations for loading a local directory, remote docker image or remote git ref. It’s all OpSource with a URI-formatted identifier. Supported URL schemes are local://, docker-image://, git://, https://

[timid gust]

Interesting.

[royal wigeon]

Top-level op:

// Op represents a vertex of the LLB DAG.
message Op {
    // inputs is a set of input edges.
    repeated Input inputs = 1;
    oneof op {
        ExecOp exec = 2;
        SourceOp source = 3;
        FileOp file = 4;
        BuildOp build = 5;
    }
    Platform platform = 10;
    WorkerConstraints constraints = 11;
}

Source op:

// SourceOp specifies a source such as build contexts and images.
message SourceOp {
    // TODO: use source type or any type instead of URL protocol.
    // identifier e.g. local://, docker-image://, git://, https://...
    string identifier = 1;
    // attrs are defined in attr.go
    map<string, string> attrs = 2;
}

Note this doesn’t map 1-1 to the corresponding Go API

Which I am more familiar with

For example here’s the llb.Git() call which is roughly equivalent to our op.#FetchGit. You can see how it maps git-specific parameters do the more generic llb “source” op: https://github.com/moby/buildkit/blob/be8ab28a0ab1f7eafa4c551a86d5aafaed7c99c2/client/llb/source.go#L210-L287

I’m wondering if we should target the Go API, or underlying LLB format, in our CUE bindings? cc @plush compass @faint cedar @flat nacelle

I guess the LLB format is more stable by necessity… But some aspects of the proto API seem awkward, with the Go API hiding some of the awkwardness… For example llb.Git() takes a remote which can be eg. ssh:// or https://. But SourceOp must prefix all git remotes with git:// so how where does the information go about the original scheme? The answer seems to be “don’t look to closely” 😉