#`directory.glob()` throwing an error complaining about "lstat ... no such file or directory"

1 messages · Page 1 of 1 (latest)

grizzled plover
#

I'm trying to

  1. Load data into the container with with_directory()
  2. Run some command (eg compile)
  3. Get the results of the command with directory() / glob()

But I'm getting exceptions where it looks like the path being generated by the glob() command is duplicating the path, something like "lstat /tmp/buildkit-mount2203948190/SOME_DIR/SOME_DIR: no such file or directory". Here is a super cut down example demonstrating the problem:

import sys

import anyio

import dagger


async def main():
    async with dagger.Connection(dagger.Config(log_output=sys.stderr)) as client:

        build_directory = client.host().directory(".")
        ctr = await (
            client
            .container()
            .from_("busybox")
            .with_directory("/build", build_directory)
            .with_exec(["echo", "hello world"], redirect_stdout="/build/output.txt")
        )

        output_directory = ctr.directory("/build")
        for filename in await output_directory.glob("**/*.txt"):
            print(filename)

anyio.run(main)
#

The exception when running this example:

16: exec echo hello world DONE
Traceback (most recent call last):
  File "/example/.venv/lib/python3.10/site-packages/dagger/client/_core.py", line 139, in execute
    result = await self.conn.session.execute(query)
  File "/example/.venv/lib/python3.10/site-packages/dagger/client/_session.py", line 129, in execute
    return await (await self.get_session()).execute(query)
  File "/example/.venv/lib/python3.10/site-packages/gql/client.py", line 1639, in execute
    raise TransportQueryError(
gql.transport.exceptions.TransportQueryError: {'message': 'resolve: resolve : lstat /tmp/buildkit-mount2203948190/build/build: no such file or directory', 'path': ['container', 'from', 'withDirectory', 'withExec', 'directory', 'glob']}
muted mantle
#

Yeah, glob seems to have some problems.

#

Which version do you have?

#

dagger version

grizzled plover
#

Originally saw the same issue with 0.9.5. Upgraded to dagger v0.10.2 (registry.dagger.io/engine) linux/amd64 and seeing the same issue

muted mantle
#

Yeah, I'm trying to debug.

#

I seem to be detecting a conflict with .venv. Do you have that dir in .?

grizzled plover
#

hmm... I do, yes

#

oh, interesting

#

it worked when I moved into another directory

#

ok, so it isn't that there is a .venv... or a hidden directory. Appears to not like any subdirectories

#

ok, only if the subdirectory contains a file that matches the glob, so something like subdir/file.bin is ok, but subdir/test.txt fails the same way (since the glob is **/*.txt)

muted mantle
#

There has to be something else at play. There's tests covering glob getting matching files recursively. I think the problem is in the path of the .directory("/build").

#

Try with ctr.directory("").glob("**/*.txt")

#

This works:

import anyio

from dagger import dag
import dagger


async def main():
    async with dagger.connection():
        d = dag.directory().with_directory("build", dag.host().directory("."))
        for filename in await d.glob("**/*.py"):
            print(filename)


anyio.run(main)

But this doesn't:

import anyio

from dagger import dag
import dagger


async def main():
    async with dagger.connection():
        d = dag.directory().with_directory("build", dag.host().directory("."))
        for filename in await d.directory("build").glob("**/*.py"):
            print(filename)


anyio.run(main)
grizzled plover
#

yeah, but the first one doesn't capture any changes made in the container (eg the "compiling" step), only items which existed in the original host directory

muted mantle
#

Doesn't matter, I'm just reproducing the issue in "glob". Doesn't matter where the directory comes from, it doesn't like it if the parent directory is a subpath.

grizzled plover
#

sorry, I just mean that I don't think the two examples are equivalent.

#

and that the weirdness in the glob() only seems to happen when "rebinding" the directory (not sure what the correct terminology is), which I believe is needed to see any new files not added from the host

muted mantle
#

I'm not suggesting a change for your code. As I said, I'm just trying to figure out what the issue with glob() is.

grizzled plover
#

ah, ok 🙂

muted mantle
muted mantle
#

Yeah, the path that is joined, is the same that's later recursed, but then it's joined again:

  • First time (join dir path with glob path and remove first slash): join("/build", ".") == "build"
  • Found .venv in build with a .txt somewhere!
  • Recurse glob (join dir path with new glob path from previous step): join("/build", "build/.venv") == "build/build/.venv"
  • Error: build/build doesn't exist!
grizzled plover
#

Should I open a bug on github?

muted mantle
#

Yes please. I have a fix, I'll submit the PR tomorrow 🙂

muted mantle
grizzled plover
#

sorry, didn't see this until this morning. Thanks for handling