#ContainerDagOp

1 messages · Page 1 of 1 (latest)

bold laurel
#

I had a quick go at trying it with my branch.

I'm working on container.WithSymlink, which should call into directory.WithSymlink. There's a mismatch between FSDagOp and ContainerDagOp.

I wonder if we'll need to change the callback that's passed tocontainer.writeToPath from func(dir *Directory) to func(ctx context.Context, dir *Directory) where this inner ctx has a FSDagOp in it instead?

summer hollow
#

ahhh, yes, there's a disparity here, so

#

i'm around if you want to sync 😄

#

so, what we should do is have core's Container.WithSymlink not take the dagop itself (or take some dagop interface or something that's generic between ContainerDagOp and DirectoryDagOp, it could extend the CustomOp type, and add things like Group, Data, etc) - then the relevant schema calls just do the right thing.
this might take a little bit of fidgetting with how dagops go through the context's WithValue to get it to work, but conceputally this should be fine

#

to get it set so that Container's dagop read it, you just write to the corresponding Result at the right point on Container (which should have been determined by locatePath)

#

so the writeToPath function we have would now need to take the returned Directory, and then set the Directory.Result to the Container.FSResult or Container.Mounts[].Result (depending on what mount was actually selected)

summer hollow
#

^ @bold laurel 🎉
i finally put together a proper op

#

and also added a wip demo commit to show how it can be used (i was using it for testing)

#

essentially, you should be able to implement a variation of locatePath that would select the right mount+state combo

#

there's also no withexec stuff in there, so it should "just work"

#

there's still a few XXX things I need to come back to tomorrow, but i need to head to the pub now 🍻

bold laurel
#

sweet, enjoy your night!

summer hollow
#

if you try and use it and get horrible panics from inside buildkit then, uh, just lemme know ❤️
i've spent all day trying to debug things like bugs in goroutines spawned by io.Copy

bold laurel
#

it seemed to work at least once, but then I started to consistently get an error like:

! failed to mount /tmp/buildkit-mount3268815237: [{Type:overlay Source:overlay Target: Options:[index=off
! lowerdir=/var/lib/dagger/worker/snapshots/snapshots/25507/fs:/var/lib/dagger/worker/snapshots/snapshots/25496/fs:/var/lib/dagger/worker/snapshots/snapshots/38/fs:/var/lib/dagger/worker/snapshots/snapshots/36/fs:/var/lib/dagger/worker/snapshots/snapshots/30/fs:/var/l
! ib/dagger/worker/snapshots/snapshots/29/fs:/var/lib/dagger/worker/snapshots/snapshots/28/fs redirect_dir=off]}]: no such file or directory

I cleared my cache and it's working again; It could very well be an unrelated issue from when I was using a different branch, since I've run it probably 20-30 times now while working on tests, and so far no issues. , nicely done!

Here's my work so far: https://github.com/dagger/dagger/compare/main...alexcb:dagger:with-symlink-on-jedevc-container-dagop?expand=1

I still need to unify some code between the core.container and core.directory WithSymlink methods, and add a test under the directory side for cases where os.MkdirAll is needed.

GitHub
Comparing dagger:main...alexcb:with-symlink-on-jedevc-container-dag...

An open-source runtime for composable workflows. Great for AI agents and CI/CD. - Comparing dagger:main...alexcb:with-symlink-on-jedevc-container-dagop · dagger/dagger

summer hollow
#

aha looks good!
one test case to be aware of is making a symlink inside a mounted directory - the symlink should be created in the mounted directory (so removing the mount removes the symlink, or inspecting the mount gives the symlink)
so instead of always selecting op.Inputs()[0] (which is the first input, the root mount), we'd need to find the right mount

#

you can see this logic in writeToPath, we'd need that equivalent

bold laurel
#

I was wondering about that case, I tried withDirectory but not withMountedDirectory.

summer hollow
bold laurel
summer hollow
#

starting to look really good! ❤️

#

at some point you asked about our use of op.Group

#

I can't find where, but figured i should answer - so best i can tell, this is related to buildkit's lazy pulling of container images

#

where buildkit will try and avoid pulling an image until it's needed - but because it's been deferred, it means that you might need a session to pull it in, since the registry might require auth

#

that's why it's ending up everywhere - and also why it's okay in local+git sources, because you don't need to pull anything.