#Seeking some clarification on how Host().Directory() is resolved.

1 messages · Page 1 of 1 (latest)

feral mica
#

Hi there,
I've got a use case where I compile a jar via Maven, extract it from the container, then run a DockerBuild in the directory I exported. The behavior, however, is confusing me.

The code looks a bit like this

src := daggerClient.Host().Directory("./some-mvn-subproject")
mvnCache := daggerClient.CacheVolume("some cache value")
daggerClient.Container().From(imageString).
        WithWorkdir("workspace").WithDirectory(".", src).
        WithMountedCache("/var/.m2/repository", mvnCache).
        WithExec([]string{"mvn", "package"})
_, err := mvnImage.Directory(".").Export(context.Background(), "./some-mvn-subproject")

at this point, great. I have the jar in the some-mvn-subproject/target directory as expected. I'm going to build a Dockerfile now that does something like

COPY target/my-jar.jar target/

but when running

buildContainer := daggerClient.Host().Directory("./some-mvn-subproject").DockerBuild()

(assuming there is a Dockerfile at some-mvn-subproject/Dockerfile). The issue is I get

failed to calculate checksum of ref <...>::<...>: "/target/my-jar.jar": not found

I originally thought I had just pointed to incorrect directories, but interestingly, if I immediately re-run, it works. In other words, if the jar exists at that location at the start of execution, it works. It's almost as if the dagger client is using a snapshot of the host directory from before the jar was exported (which I don't think is true).

Does this look familiar to anyone or am I perhaps misunderstanding the functionality here? Perhaps I'm just using context incorrectly?

rare dagger
#

Hi @feral mica , exporting host directories can’t be used as intermediary steps in your pipeline - those are only created at the end of executing the pipeline

#

you can just reference the directory directly, from one container to the next, without the intermediary export and reimport to the host

#

It's almost as if the dagger client is using a snapshot of the host directory from before the jar was exported (which I don't think is true).

That's exactly right, the same snapshot of a host directory will be used for the entire session. The snapshot is used for content-addressing and caching. This makes host directories unsuitable for composition inside your dagger session.

# 
// Load source
src := daggerClient.Host().Directory("./some-mvn-subproject")
mvnCache := daggerClient.CacheVolume("some cache value")
// Run 'mvn package'
mvnPackage := daggerClient.Container().From(imageString).
        WithWorkdir("workspace").WithDirectory(".", src).
        WithMountedCache("/var/.m2/repository", mvnCache).
        WithExec([]string{"mvn", "package"}).
        Directory(".")
// Run docker build
buildContainer := mvnPackage.DockerBuild()
feral mica
#

Aha! Ok great, I appreciate the clarification. I was looking to refactor the code into utilizing directory options, was just trying to get something quick up and running. Will do that refactoring now

#

Thanks for the help and quick response!

rare dagger
#

No problem. Of course you can also export any of those directories to the host at any time, as long as you think of them as "leafs" or "exit nodes", not referenced elsewhere in the DAG. I just removed the exports in the code snippet for clarity

pure oar
#

cc @regal fractal I can't recall if we mention this in our docs, but definitely not the first time I see users being caught by this

rare dagger
feral mica
#

I have one more follow-up from this.

Your suggestion has worked as intended, however, I don't have a good solution to image scanning. After our image is built, I want to run it through Sysdig. For our company, the provided input is a tar (similar to what you'd get from docker save).

My initial approach was going to be something like

buildContainer := mvnPackage.DockerBuild()
_, err := buildContainer.Export(context.Background(), "oci-image.tar")
hostWithTar := daggerClient.Host().Directory(".")
daggerClient.Container().From("sysdig-image").
       WithWorkdir("workspace").WithDirectory(".", hostWithTar).WithExec("<some commands to scan the tar">).Stderr(context.Background())

where the oci-image.tar is just passed in from the host and referenced. Obviously I better understand now that this doesn't work, but I don't necessarily see a good way of packaging this buildContainer up into a tar that can be scanned prior to publication.

My workaround for now is to just run this scanning on the host itself and concede that it needs access to the relevant commands, but curious if I missed documentation somewhere or if there had been thought towards use cases like this

pure oar
feral mica
#

our interface actually makes an API call (as a way to not need all users to have a sysdig binary), so it's uploading the tarball to a server that then executes the scan, but that sounds like there's some potential.

Can I upload it to the local registry, grab the image id, and fire a docker save equivalent in the sysdig image to produce the tar within the dagger session?

rare dagger
#

We are planning a Container.Archive which would produce the tarball from any container. But there are implementation challenges so, not done yet

#

there is container.export which will do that but on the host filesystem, so not great if you want to continue with the scanning in dagger

#

One more thought: I wonder if you could run an ephemeral registry jn dagger, using a service container, then push to it? That was planned in the container networking design but Ii don’t know if we ever got to it (also a question for @stark star )

stark star
#

I think what you can do, though, is Container.export to a different directory on the host that hasn't been synced to Dagger yet, like a tmpdir

#

pretty sure we do something like that in our own test suite

feral mica
#

ah, that's an interesting idea. So if I have

image-library/
workspace/
 -> repo/
    -> ...

and I call daggerClient.Host().Directory("workspace"), I can later on take a snapshot of image-libarary after exporting the tar there? I can play around with that

#

the container.Export approach is what I've gone with for now and I've just made that work, so the implementation is otherwise complete and we're not blocked at the moment. By this I mean I just perform the Sysdig scan on the host itself with the exported image outside of Dagger