Hi, I upgrade my dagger engine on 0.18.6 | Dagger | Page 1

bright drum May 7, 2025, 9:02 PM

#

It's from some type you have defined. Do you happen to have a type declared with just list instead of also the subscript like list[str]? Otherwise could you share some of your code?

vital roost May 12, 2025, 8:28 AM

#

Hi, I take a look on my code, the issue is on a submodule:

@object_type
class S3:
    base_ctr: Annotated[dagger.Container, Doc("Base container to use for all functions")]
    bucket_name: Annotated[str, Doc("The targeted bucket name")]
    src: Annotated[dagger.Directory, Doc("Local folder to use for s3 actions")]

    @function
    async def sync(self, path: str | None) -> str:
        s3_uri = f"s3://{self.bucket_name}/" if path is None else f"s3://{self.bucket_name}/{path}"
        return await (
            self.
            base_ctr.
            with_directory("/aws/src", self.src).
            with_exec(["aws", "s3", "sync", "/aws/src/", s3_uri, "--follow-symlinks", "--delete"]).
            stdout()
        )

When I'm calling the sync function I have the previous error.
If I remove the path parameter like that:

    @function
    async def sync(self) -> str:
        path = None
        s3_uri = f"s3://{self.bucket_name}/" if path is None else f"s3://{self.bucket_name}/{path}"
        return await (
            self.
            base_ctr.
            with_directory("/aws/src", self.src).
            with_exec(["aws", "s3", "sync", "/aws/src/", s3_uri, "--follow-symlinks", "--delete"]).
            stdout()
        )

it's working

#

I just try to do with a default parameter:

    async def sync(self, path: str = "/") -> str:
        s3_uri = f"s3://{self.bucket_name}{re.sub(r'(?<!:)//+', '/', path)}"
        return await (
            self.
            base_ctr.
            with_directory("/aws/src", self.src).
            with_exec(["aws", "s3", "sync", "/aws/src/", s3_uri, "--follow-symlinks", "--delete"]).
            stdout()
        )

and I have the same error ! unhashable type: 'list'

vital roost May 12, 2025, 12:35 PM

#

I just tried to call my function directly and it's working so not sure why we a recent version I have this error

#

My aws module looks like:

@object_type
class Aws:
    aws_cli_version: Annotated[str, Doc("The aws cli version to use")] = field(default="2.18.5")
    __base_container: dagger.Container = dataclasses.field(init=False)

    def __post_init__(self):
        self.__base_container = (
            dag.container()
            .from_(f"amazon/aws-cli:{self.aws_cli_version}")
        )

    @function
    async def s3(self, bucket_name: str, src: dagger.Directory) -> S3:
        """Returns S3 functions"""
        return S3(base_ctr=self.__base_container, bucket_name=bucket_name, src=src)

Then S3 is the class I shared previously

#

this aws module is called in a Front module which is building js front app then deploy the front in a s3 bucket

#

the end function which is calling my aws module:

    @function
    async def deploy(
            self,
            account: Annotated[str | None, Doc("The aws account name")],
            pipeline_id: Annotated[str | None, Doc("Kind of namespace between different projects")]
    ) -> str | None:
        """Deploy a frontend application to a s3 bucket and invalidate the cache in cloudfront"""
        if not account:
            account = self.account

        if not pipeline_id:
            pipeline_id = self.pipeline_id

        awscli = dag.aws().with_profile(name=self.account)
        secret_manager = dag.secret_manager().aws().with_profile(name=self.aws_profile)

        if self.aws_creds:
            secret_manager = secret_manager.with_credentials_folder(aws_folder=self.aws_creds)
            awscli = awscli.with_creds(src=self.aws_creds)

        deploy_output = await (
            awscli.
            s3(
                bucket_name=await (
                    secret_manager
                    .get_secret(
                        f"CI/{account.upper()}_{pipeline_id.upper()}_S3_BUCKET_NAME").
                    plaintext()
                ),
                src=self.__code.directory(self.__build_path)
            ).
            sync()
        )

        invalidation_log = await (
            awscli.
            cloudfront(
                distribution_id=await(
                    secret_manager.
                    get_secret(
                        f"CI/{account.upper()}_{pipeline_id.upper()}_CF_DISTRIBUTION_ID"
                    ).
                    plaintext()
                )
            ).
            invalidate_cache(
                run_id=await self.__code.directory(self.__build_path).digest()
            )
        )

        return deploy_output + "\n\n================================\n\n" + invalidation_log

#

what is strange is my cloudfront submodule is working and have this signature:

@object_type
class Cloudfront:
    base_ctr: Annotated[dagger.Container, Doc("Base container to use for all functions")]
    distribution_id: Annotated[str, Doc("The targeted cloudfront distribution")]

    @function
    async def invalidate_cache(
            self,
            paths: Annotated[list[str] | None, Doc("Define http paths to purge in the CDN")] = None,
            run_id: Annotated[str | None, Doc("Id which define if the invalidation should run or not, it's like a cache burster")] = None,
    ) -> str:

#

I just tried to set None as default value for my sync function in S3 submodule

#

but I still have the issue

bright drum May 12, 2025, 12:45 PM

#

Thanks for sharing, @vital roost, nothing immediately pops out to me. Much easier if we can get a simple reproducing example. Can you try turning on debug logging to see if it helps pin point more?

vital roost May 12, 2025, 12:45 PM

#

yes I do that now

bright drum May 12, 2025, 12:46 PM

#

You said it's a submodule. Does it mean the one that's erroring is being called from another Dagger module, but if you run it directly you don't get the error?

vital roost May 12, 2025, 12:47 PM

#

right

#

front is calling aws (which has multiple object_type like s3 cloudfront ...)

bright drum May 12, 2025, 12:47 PM

#

That could mean it's in the codegen/client bindings.

bright drum May 12, 2025, 12:49 PM

#

vital roost `front` is calling `aws` (which has multiple `object_type` like `s3` `cloudfront...

Are you able to strip them down to bare minimum just to help zone in on the issue?

#

Or if you think you understand what reproduces it, can you create a new basic module that gives the same error?

vital roost May 12, 2025, 12:50 PM

#

I have this debug log:

📎 message.txt

bright drum May 12, 2025, 12:51 PM

#

Ok, that's helpful.

vital roost May 12, 2025, 12:52 PM

#

the issue is only on the s3 part, I'm not sure what do you want by creating a basic module or strip down the module

bright drum May 12, 2025, 12:52 PM

#

Give me a moment to analyze what you've shared

vital roost May 12, 2025, 12:52 PM

#

ok

bright drum May 12, 2025, 12:55 PM

#

On this block:

        deploy_output = await (
            awscli.
            s3(
                bucket_name=await (
                    secret_manager
                    .get_secret(
                        f"CI/{account.upper()}_{pipeline_id.upper()}_S3_BUCKET_NAME").
                    plaintext()
                ),
                src=self.__code.directory(self.__build_path)
            ).
            sync()
        )

Can you try passing simple values to s3()? Like a hardcoded string for bucket_name and a dag.directory() for src?

#

The point is to see if you get the same error.

#

If you do I'll want to look at awscli next.

#

Still have base_ctr=self.__base_container in s3 I see. Can I ask why the double __ prefix?

vital roost May 12, 2025, 12:58 PM

#

ok

#

double underscore in python doesn't mean the attribute is private ? or it was in python2 ?

bright drum May 12, 2025, 1:05 PM

#

You only need a single _ to "signal" it as private in Python (by convention). The double __ mangles it so it can't be use by a subclass, but this is usually used for some specifc conditions. In the context of a Dagger module, if the idea is not to expose as a function in the Dagger API you can just not use dagger.field and use dataclasses.field (if needed) instead.

The reason I point this out is that mangling is not tested so I'll need to check how it's being handled. The issue in my mind has to do with serialization/deserialization. You see, when you call a dagger function through the API, be it from the CLI or from your code using dag, each function is called in its own runtime container. It's not the same Python process. You can see in the logs you shared what's the state that the engine is passing to call "deploy". That function doesn't have any more state outside of that.

#

Does parent_json contain the state that you expect?

#

That's the state for the "Front" object when "deploy" is called.

vital roost May 12, 2025, 1:07 PM

#

I just tested with the hardcoded bucket name + empty folder I have the same error, do you want debug logs ?
I can switch __ to _

#

what do you want parent_json ?

bright drum May 12, 2025, 1:08 PM

#

vital roost I just tested with the hardcoded bucket name + empty folder I have the same erro...

I'm just wondering about that container. Does parent_json have a value for you there?

vital roost May 12, 2025, 1:09 PM

#

I don't know what is parent_json

bright drum May 12, 2025, 1:10 PM

#

bright drum I'm just wondering about that container. Does `parent_json` have a value for you...

I mean here, the s3 function expects it:

    @function
    async def s3(self, bucket_name: str, src: dagger.Directory) -> S3:
        """Returns S3 functions"""
        return S3(base_ctr=self.__base_container, bucket_name=bucket_name, src=src)

bright drum May 12, 2025, 1:10 PM

#

vital roost I don't know what is `parent_json`

Look at the beggining of the debug logs you shared:

✘ .deploy: String 48.4s
[DEBUG] dagger.client._session: Configuring shared connection to GraphQL server                   [DEBUG] dagger.client._session: Establishing client session to GraphQL server                     [DEBUG] dagger.mod._module: invoke => {'parent_name': 'Front', 'parent_json': '[...]', 'name': 'deploy', 'input_args': '{}'}                                                                     [DEBUG] dagger.mod._module: structured args => {'account': None, 'pipeline_id': None}             [DEBUG] dagger.mod._module: func => <Signature (account: Annotated[str | None, Doc('The aws account name')], pipeline_id: Annotated[str | None, Doc('Kind of namespace between different projects')]) -> str | None>
[DEBUG] dagger.mod._module: input args => {}

#

Specifically this line:

[DEBUG] dagger.mod._module: invoke => {'parent_name': 'Front', 'parent_json': '[...]', 'name': 'deploy', 'input_args': '{}'}

#

Check if parent_json contains the state you expect when deploy function executes.

vital roost May 12, 2025, 1:12 PM

#

ok how can I check it ? because it's not one of my variables

bright drum May 12, 2025, 1:13 PM

#

Notice the field names in particular. If it's a dagger object you'll see a digest. It's not important to check that value, just that it's there.

bright drum May 12, 2025, 1:14 PM

#

bright drum I mean here, the `s3` function expects it: ```python @function async def...

Sorry this is confusing. Just noticed this is in "Aws" object, not "Front" object.

vital roost May 12, 2025, 1:16 PM

#

no problem so at the end what can I share or do ?

bright drum May 12, 2025, 1:17 PM

#

What's the shortest awscli instantiation you can try?

awscli = dag.aws()

vital roost May 12, 2025, 1:17 PM

#

I'm trying the code ny changing __base_containerto _base_container

#

yes I can I try but I did it earlier when I was trying to understand where is the error

#

and it was working without the s3 sync

#

I can retry anyway

bright drum May 12, 2025, 1:19 PM

#

Oh wait, you have a function called sync. Can you rename it? It's a possible conflict.

vital roost May 12, 2025, 1:19 PM

#

ok I try that

bright drum May 12, 2025, 1:19 PM

#

Although I don't believe it is in this case.

#

Can you simplify? Something like this:

async def debug(self) -> str | None:
    return await dag.aws().s3(bucket_name="foo", src=dag.directory()).sync()

vital roost May 12, 2025, 1:24 PM

#

renaming fix the issue

bright drum May 12, 2025, 1:24 PM

#

Ok, codegen could be confused by "sync" then. Let me see if I can make a basic repro.

vital roost May 12, 2025, 1:25 PM

#

ok

#

with previous versions between 0.13.3 to 0.16.1 it was working

bright drum May 12, 2025, 1:27 PM

#

Yeah, a change was added to codegen that could be affecting this.

#

Basic repro:
Module "front"

from dagger import dag, function, object_type


@object_type
class Front:
    @function
    async def test(self) -> str:
        return await dag.aws().sync()

Module "aws"

from dagger import function, object_type


@object_type
class Aws:
    @function
    def sync(self) -> str:
        return "hello world"

It's not the same error, but I can already see that function wasn't condegened correctly.

vital roost May 12, 2025, 1:31 PM

#

ok

bright drum May 12, 2025, 1:32 PM

#

I'll try to fix this quickly. We should have a new release this week.

#

Thanks for reporting and sticking with me while debugging!

vital roost May 12, 2025, 1:33 PM

#

Thank you for your help and no problem this is normal 🙂
Ok good news for the release

#Hi, I upgrade my dagger engine on 0.18.6