container build from dockerfile fails | Dagger | Page 1

clear wedge Mar 21, 2024, 1:23 AM

#

having Dockerfile, when i build it using docker build as usual it works. but inside dagger with dag.container.build it fails usually around docker-php-ext-install -j$(nproc) syntax. What im missing? thanks

📎 message.txt

humble peak Mar 21, 2024, 9:05 AM

#

what does the error say?

clear wedge Mar 21, 2024, 11:45 AM

#

📎 message.txt

hardy quail Mar 21, 2024, 1:30 PM

#

I am trying to reproduce locally, @clear wedge could you please share the whole dagger function that you wrote that are calling as well?

clear wedge Mar 21, 2024, 1:31 PM

#

    async def build_all(self, dockerfile: str, build_number: str) -> dict[str, Container]:
        """Build project all container images."""

        await dag.logger().log(f"Building all container images with build number {build_number}")

        git_hash = await self.git_commit_hash()

        build_args = [
            BuildArg(name="VERSION", value=git_hash),
            BuildArg(name="GIT_COMMIT_HASH", value=git_hash),
            BuildArg(name="GHA_BUILD_NUMBER", value=build_number),
        ]

        container_dict = {}
        for target in ["dev", "ci", "api", "cmd"]:
            container_dict[target] = dag.container().build(context=self.working_dir, dockerfile=dockerfile, target=target, build_args=build_args)

        await asyncio.gather(*container_dict.values())

        await dag.logger().log(f"All container images have been built with build number {build_number}")

        return container_dict```

#

use any value for the args

hardy quail Mar 21, 2024, 2:02 PM

#

I was not able to reproduce with the simplest example. I copied over your Dockerfile and ran this function:

    @function
    def build(self, dir: dagger.Directory) -> dagger.Container:
        """Returns docker container from dockerfile"""
        return dag.container().build(dir)

I call this with dagger call build --dir .

This returns the ci image from what I can tell, but gets through the j$(nproc) step without any issues.

Only difference is I have an empty framework/.env file since I don't know what your values are in there, not sure if this has something to do with it.

Are you able to get this part to work? I would start there before moving on to more advanced flow.

clear wedge Mar 21, 2024, 3:03 PM

#

umm weird, is still not working for me

hardy quail Mar 21, 2024, 3:10 PM

#

Can you tell me more about the version of dagger you’re using and what environment you’re running in?

clear wedge Mar 21, 2024, 3:18 PM

#

dagger v0.10.2 (registry.dagger.io/engine) darwin/arm64

#

running on mac

wintry frost Mar 21, 2024, 4:40 PM

#

I'm also on mac, latest version, and can't reproduce as well.

#

Doesn't error on syntax

#

@clear wedge, did you try Lev's repro? What are you passing to your dockerfile argument? The Dockerfile contents or the path?

clear wedge Mar 21, 2024, 4:47 PM

#

wintry frost <@783047641796706335>, did you try Lev's repro? What are you passing to your `do...

the path/name this is just to allow different dockerfiles in my build

#

ok i tried removing all options beside the dir and it works. so its something related to either args or dockerfile. will get back with more info after testing. thanks

wintry frost Mar 21, 2024, 4:51 PM

#

Btw returning a dict is not supported.

clear wedge Mar 21, 2024, 5:01 PM

#

i just confirm it fails when passing build args. its ok to pass it as im doing right now?

            BuildArg(name="VERSION", value=git_hash),
            BuildArg(name="GIT_COMMIT_HASH", value=git_hash),
            BuildArg(name="GHA_BUILD_NUMBER", value=build_number),
        ]```

clear wedge Mar 21, 2024, 5:30 PM

#

ok i have narrow it down to this. the build args that relies in git_hash are the ones causing the issue, the value of that variable im computing it from this function

    async def git_commit_hash(self) -> str:
        """Get the git commit hash."""

        container = (
            dag.container()
            .from_(address="alpine/git")
            .with_mounted_directory(path="/mnt", source=self.working_dir or ".")
            .with_workdir(path="/mnt")
            .with_exec(["rev-parse", "HEAD"])
        )

        return await container.stdout()```
and calling it as
```git_hash = await self.git_commit_hash()```
for some reason that fails. the value its ok as it is like a48cc89c6ed44d07a8964d36f9adabd9d7979594 but it fails. any idea why?

wintry frost Mar 21, 2024, 6:42 PM

#

Your with_mounted_directory looks strange. source= should be a Directory. I don't know what's in self.working_dir, but "." is definitely not valid.

#

But since you say you're getting a valid value here, I'm assuming you have a directory there, so let me try it for a sec.

clear wedge Mar 21, 2024, 6:47 PM

#

self.working_dir is a dir. actually pass as dagger parameter with --working-dir=.

#

please ignore the last string

wintry frost Mar 21, 2024, 6:48 PM

#

Progress 🙂 Now I can reproduce, looking into it...

#

So far the build is progressing, so I seem to have found the culprit. Gonna make a smaller repro.

clear wedge Mar 21, 2024, 7:01 PM

#

wintry frost So far the build is progressing, so I seem to have found the culprit. Gonna make...

which was?

wintry frost Mar 21, 2024, 7:09 PM

#

Extra \n in VERSION.

#

From the output of rev-parse. Need to trim that.

#

It's giving issues with a curl php extension.

#

Try it with docker build. Just add the \n to the end of the VERSION build arg you're passing in.

wintry frost Mar 21, 2024, 7:32 PM

#

Repro:

import dagger
from dagger import dag, function, object_type


DOCKERFILE = r"""
FROM php:8.3-cli

RUN apt update \
 && apt install --no-install-recommends -y libcurl4-openssl-dev

ARG FOO
ENV VERSION "${FOO}"

RUN docker-php-ext-install curl
"""


@object_type
class Dockerfile:
    @function
    def build(self) -> dagger.Container:
        return (
            dag.directory()
            .with_new_file("Dockerfile", contents=DOCKERFILE)
            .docker_build(
                build_args=[
                    dagger.BuildArg("FOO", "foobar\n"),
                ],
            )
        )

#

If not setting the build args, the default ARG VERSION="unstable\n" doesn't error. Need to investigate how dagger is adding the build args.

wintry frost Mar 21, 2024, 8:25 PM

#

Re: the syntax failure, it comes from a curl libtool template that starts like this:

#! /bin/bash

# libtoolT - Provide generalized library-building support services. Generated automatically by  (GNU $VERSION)
# NOTE: Changes made to this file will be lost: look at ltmain.sh.

So when VERSION is foobar\n, the file that it produces looks like this:

#! /bin/bash

# libtoolT - Provide generalized library-building support services. Generated automatically by  (GNU foobar
)
# NOTE: Changes made to this file will be lost: look at ltmain.sh.

You can see that the line got broken without a #, thus the following error:

/usr/src/php/ext/curl/libtool: line 5: syntax error near unexpected token `)'

It's not about $(nprojc).

clear wedge Mar 21, 2024, 8:40 PM

#

omg so there is an actual $VERSION variable been use by other tools. good catch. appreciate it

wintry frost Mar 21, 2024, 8:42 PM

#

Yeah, too general 🙂

wintry frost Mar 21, 2024, 9:04 PM

#

https://github.com/php/php-src/blob/530e0d68ebee5b21ad10311115c392072b8b4168/build/libtool.m4#L4124-L4126

clear wedge Mar 21, 2024, 9:19 PM

#

if i may. how did you got to that file? i was trying to find the bug myself 🤦🏻‍♂️

wintry frost Mar 21, 2024, 9:26 PM

#

In the Dockerfile I did RUN docker-php-ext-install curl || true so it didn't fail the pipeline, then just export to inspect locally: dagger call build file --path=/usr/src/php/ext/curl/libtool export --path=libtool (or just contents instead of export ...).

#

It was obvious when I opened the file.

clear wedge Mar 21, 2024, 9:33 PM

#

amazing debug skills, need to learn those tricks

wintry frost Mar 21, 2024, 9:41 PM

#

Smallest repro yet:

import dagger
from dagger import dag, function, object_type


DOCKERFILE = r"""
FROM bash

ARG FOO="vs\n"

RUN cat <<EOF >> /script
# Cats ($FOO)
# Dogs
echo OK!
EOF
RUN cat -n /script && bash /script
"""


@object_type
class Dockerfile:
    @function
    def build(self) -> dagger.Container:
        return (
            dag.directory()
            .with_new_file("Dockerfile", contents=DOCKERFILE)
            .docker_build(
                build_args=[
                    dagger.BuildArg("FOO", "vs\n"),
                ],
            )
        )

With BuildArg:

  ✘ [3/3] RUN cat -n /script && bash /script 0.1s
  ┃      1  # Cats (vs
  ┃      2  )
  ┃      3  # Dogs
  ┃      4  echo OK!
  ┃ /script: line 2: syntax error near unexpected token `)'
  ┃ /script: line 2: `)'

But if you comment the BuildArg:

# Cats (vs\n)
# Dogs
echo OK!

#

But this also works (raw string):

dagger.BuildArg("FOO", r"vs\n"),

#

Should Dagger be passing every build arg value raw to the docker build? 🤔

clear wedge Mar 22, 2024, 1:29 PM

#

there is any real use case for a build arg to have a line brake "\n"?

wintry frost Mar 22, 2024, 2:08 PM

#

That depends entirely on anyone’s use case. Hard to say it’s never desired.

clear wedge Mar 22, 2024, 2:26 PM

#

ok so then the answer is leave it as it is 😉

wintry frost Mar 22, 2024, 3:13 PM

#

Question can be do we make sure it's always raw or not? However, docker build does seem to trim. cat'ing a file that terminates in multiple new lines in a --build-arg you can see the ARG in the Dockerfile ends up without any new line at the end.

#container build from dockerfile fails