#Container.Directory() cannot find existing directory

1 messages · Page 1 of 1 (latest)

gray harness
#

Hi! It's me again, back with another issue I could need help with.

I 'm running a Dagger pipeline with a Python container, which has a service binding to a Clickhouse database container (https://hub.docker.com/r/clickhouse/clickhouse-server/). So far so good, everything works as expected.

Now what I want to do is create a backup of a database in Clickhouse to "local disk" (the Clickhouse container), and export it to the host (to restore and debug).
Creating the backup from within Clickhouse works fine, and when I run await ctr.with_exec(["sh", "-c", "ls"]).stdout() on the Clickhouse container, the backup folder foobar is listed (output shortened for post length):

[...]                                                                                   
┃ entrypoint.sh                                                                                                   
┃ etc                                                                                                             
┃ foobar                                                                                                          
┃ home
[...]

When I run await ctr.directory(".").entries() though, the directory is not listed:
['bin', 'boot', 'dev', 'docker-entrypoint-initdb.d', 'entrypoint.sh', 'etc', 'home', 'lib', 'lib32', 'lib64', 'libx32', 'media', 'mnt', 'opt', 'proc', 'root', 'run', 'sbin', 'srv', 'sys', 'tmp', 'usr', 'var']

Could this be caused by some virtual filesystem shenanigans, as mentioned in this post https://discord.com/channels/707636530424053791/1139358960927908013 ?

EDIT: I created a minimal example to reproduce my issue: https://gist.github.com/apokolokyntosis/e83813de64f130d981e853b139859cc8

Thanks in advance!

Gist
Dagger

Dagger. GitHub Gist: instantly share code, notes, and snippets.

tame yew
#

Hey @gray harness 👋 That's an interesting one. Could you give a snippet of your code just to have a better idea of what's happening? Also ls -la might have some clues too, but just guessing at this point

gray harness
#

Hey, thanks for the reply! Sure, I'll put something together really quick!

gray harness
#

Trying to a minimal Example running, but I keep hitting the following Error:

dagger.SessionError: Failed to start Dagger engine session: Command '/root/.cache/dagger/dagger-0.8.8 session --label dagger.io/sdk.name:python --label dagger.io/sdk.version:0.8.8' returned non-zero exit status 1.                 
█ [2.14s] ERROR exec python main.py
┃ 1: connect               
┃ 1: > init                
┃ 1: starting engine               
┃ failed to list containers: exec: "docker": executable file not found in $PATH                
┃ 1: starting engine [0.               
┃ 1: connect ERROR: new client: failed to run container: : exec: "docker": executable file not found in                
┃ Error: new client: failed to run container: : exec: "docker": executable file not found in $PATH
#

Installed Dagger binary is v0.8.8, running engine is v0.8.8 and python sdk is also ==0.8.8

tame yew
#

Can you confirm you have a docker runtime in your environment?

gray harness
#

You mean like this?

❯ docker version
Client:
 Version:           24.0.5
 API version:       1.43
 Go version:        go1.19.12
 Git commit:        %{shortcommit_cli}
 Built:             Sun Aug 27 16:48:40 2023
 OS/Arch:           linux/amd64
 Context:           default

Server:
 Engine:
  Version:          24.0.5
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.19.12
  Git commit:       %{shortcommit_moby}
  Built:            Sun Aug 27 16:48:40 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.19
  GitCommit:        
 runc:
  Version:          1.1.8
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:
#

The thing is, I was running Dagger pipelines all day long for the last few weeks

#

I wanted to doublecheck whether the versions matched, and set up a new python venv, and I wasn't able to run a pipeline since

tame yew
#

yeah I was just asking because failed to list containers: exec: "docker": executable file not found in $PATH sounds like dagger can't find docker

gray harness
#

Yes, makes sense!

tame yew
#

if you can share the pipeline code I can try it to see if it's something in the code or the environment

gray harness
#

Yeah sure! I was just trying to set up a minimal example for you, since the original pipeline contains customer code. Sec

tame yew
#

Ah, right because the pipeline runs itself (main.py), but docker isn't in the python:3.11-slim-buster image

#

so dagger+docker are working, but the pipeline is also trying to run a dagger pipeline within itself. Which I guess would be infinitely recursive so it's probably good that it didn't work

gray harness
#

Oh jesus. I'm an Idiot. I wanted to run db.py in the pipeline...

#

Thank you! This was the reason obviously

gray harness
#

I updated the gist. You should be able to see the behavior now

tame yew
#

Thanks, I'll have a look!

tame yew
#

Confirmed I can reproduce the issue and I'm not sure why it's happening.

@plush ocean any guesses? The above gist is a working repro for me. Basically trying to directory().export() from a service container. The directory to be exported shows up in an ls, but the .directory() fails with {'message': 'lstat /tmp/buildkit-mount622740753/foobar: no such file or directory',

plush ocean
#

I recall we discussed about it somewhere in #1030538312508776540 some time ago. The current workaround is to put those files in a volume and then copy them back to a container to export them

tame yew
#

Ah got it. So the services v2 about to be released will help here then

plush ocean
#

now with the upcoming 0.9 release, it will be possible to handle this use-case better since the service container lifecylce can be managed better

gray harness