Image export to local runtime w/o writing file

[limpid hill]

Currently, we have a workflow that consists of

• docker build
• docker save
• ctr import

I'd love to be able to streamline this process with Dagger, and we have the first two with container.Export (https://pkg.go.dev/dagger.io/dagger#Container.Export)

What I'd like to be able to do is get the bytes and use io.Copy or similar to ship the bytes directly to containerd.

Is this possible today? Do we need another SDK function?

[tidal plinth]

AFAIK not currently possible. The main challenge is that because the core API is GraphQL, it's not trivial to just stream a []byte given that it's not supported by the protocol. We'd have to encode/decode it somehow and it'd still not be streamed. cc @strange gazelle @lean bough

[strange gazelle]

Yeah definitely doable, but requires each SDK support streaming, so just an effort multiplied by number of SDKs

[lean bough]

Hm I'm not sure streaming is the blocker here - I've actually really wanted the ability in dagger to output to an arbitrary containerd content/image store. Buildkit can kind of do this already, there's a session attachable for it

But doing this would require a new core type I guess?

One of the issues (and the reason I've not actually implemented this myself) is I don't think dockerd exposes the containerd content API externally, so you can't load images directly like that

Though we could fallback to doing a "docker load"

TL;DR - I don't think we necessarily need streaming to solve the problem, we need a general way to "load" images, in addition to exporting them. The loading should support "docker load", but also ideally would support arbitrary containerd content stores.

I'll open this into an issue

[lean bough]

👋 wrote some of this up in https://github.com/dagger/dagger/issues/8025

[tidal plinth]

I think streaming could / would be useful for a bunch of other cases also, particularly to improve some APIs like the File one so contents can be streamed.

In this particular case, for example, if we could make File.Contents(ctx) an io.Reader (and equivalent in the other SDKs), I could just do docker.Load(ctx, container.AsTarball().Contents(ctx)).

don't think dockerd exposes the containerd content API externally, so you can't

you could connect to the containerd.sock docker uses directly and call the API there I'd assume

[lean bough]

sadly you can't do that

there's a proposal for it: https://github.com/moby/moby/issues/44369

[lean bough]

mm agreed, i'd like it for this as well, but, i'm also not 100% sure how we would do this for graphql.
one option - we could have a special "stream" type that addresses content in a content store by digest - then the client can pull that using an http/grpc/etc api

then each language uses it's own primitives to do that http request as a stream

[tidal plinth]

😬 I took a guess but as you're sharing, doesn't work 😦

[lean bough]

alternatively, we implement a Stream type with read/len/etc fields that are still graphql-like, and then allow each sdk to wrap that into something more idiomatic

[tidal plinth]

yeah, I was imagining somehing along these lines also. Since GraphQL's transport is http, I guess that for these specific operations, we can just hijack the underlying GraphQL TCP / HTTP connection and do the streaming at that level

[lean bough]

if we can avoid hijacking i would be very happy 🙂

i think we could just serve it at another http endpoint

since all the graphql queries go to one place (i think)

[tidal plinth]

yep, this also works

[lean bough]

hmmm issue with doing the content addressable store - it doesn't support content that is currently still being produced, e.g. if you wanted to stream service logs

i suppose you could rectify that by just having it not be content addressable, and just random id 🙂

but then you do lose some cacheability, which would have been nice (if you were just moving the stream around and not actually reading from it)

[tidal plinth]

how about if we use Dagger's ID's for this? i.e: dag.Container().File("/foo").ID() could be the content addressable ID of its contents?

[lean bough]

mm, maybe some of the content-ids would be dagql-ids (like Files), but others might not be (like Service.Stdout, if we had that)

yeah we could totally do this

[tidal plinth]

https://tenor.com/bmCv7.gif

[lean bough]

it's definitely a large t-shirt task though imo, it requires doing some buildkit shenanigans so we can get an open file object to a file in a ref, it then needs some dagql plumbing to create like a StdoutData or something like that that returns a stream object - then we'd want to have stream objects be nicely served by a new component as part of dagger server (?), and finally some sdk magic over the top to do make the http requests for us and turn them into native stream objects