Pipeline not starting, maybe legacy?

[lusty cape]

Hi dagger team, i have some pipelines created using python sdk v0.11.7
They run once a month. Out of the blue i'm getting this error: "error: parse selections: parse field "args": __Field.args has no such argument: "includeDeprecated"
Full trace attached.

They run on AWS CodeBuild
Image: aws/codebuild/standard:6.0
Environment type: Linux Container
Trace: https://dagger.cloud/valen-spi3pl/traces/8840a852b910105792c33edc665c60db

When running them locally, the dagger engine starts, connects, but the pipeline doesnt start. https://dagger.cloud/valen-spi3pl/traces/145193a9a0057e9fa58baa69e27e270c

Any hint? Thanks a lot!

[fossil spoke]

Hey there! Can you verify your Dagger CLI hasn't accidentally upgraded to a newer version?

[lusty cape]

hola marcos! nope, im not using the cli at all, the pipeline starts programatically, with anyio.run(pipeline):

    async def build(self, config: CRUDDeployments):
        dagger_config: dagger.Config = dagger.Config(log_output=sys.stdout)
        async with dagger.Connection(dagger_config) as client:
            # create container
            container = await self.create_service_container(client, config)

            # install dependencies and source code
            container = await self.install_dependencies(config.container, container, client)

            # run tests
            if config.pipeline.steps.get('test') and config.pipeline.step_config:
                container = await self.run_tests(config, container)

            # save initial environment variables
            container = await self.set_initial_environment(config, container)

            # add start command
            container = container.with_default_args(args=config.container.default_args)

            # publish the image built by dagger
            if config.pipeline.steps.get('publish'):
                container = await self.publish_service_image(client, config, container)

            # export tarball
            if config.pipeline.export_tar:
                logger.debug('Exporting tarball...')
                await container.export(f'{config.container.name}-image.tar')

            # update ECS service
            if config.pipeline.steps.get('update'):
                logger.debug('Updating ECS service...')
                container = await self.update_ecs(config, container)

            await container.stdout()
            return container

[fossil spoke]

Perfect. When invoked via the SDK directly, the SDK downloads a dagger CLI to start the engine behind the scene

What I think it's happening is that the CLI being downloaded is a newer one for some reason

[lusty cape]

i wanted to avoid refactor to use the cli, do you know if there's a workaround to have them working again as they were?

*are

[fossil spoke]

I can't recall on the top of my head but I think there's an env variable you can set to force it to use the proper one. cc @shut agate @high willow since I'm AFK now

[fossil spoke]

Sorry about the break. A lot happened since 0.11.7 to improve backwards compatibility 🙏

[lusty cape]

i couldn't keep up with the updates yet, but i was surprised it stopped working completely

[fossil spoke]

Does it happen in your local machine also?

[lusty cape]

yep, the second trace is from my local setup

[high willow]

Did you happen to bump any dependencies in your python code recently? Or maybe don't have them all pinned? This issue (and the related fix) sound possibly related

If so, pinning gql to ~=3.5.3 should work around the issue until you upgrade to dagger v0.18.16+: https://github.com/dagger/dagger/issues/10885#issuecomment-3195968790

[lusty cape]

i tried updating to 0.18.16 but the issue remains, let me do a run and give you the trace

https://dagger.cloud/valen-spi3pl/traces/1d3a1694b8d063bbade13f5c0067431c

[high willow]

Is your python code spitting out the same error as before for this trace? The trace itself actually looks ok, like it succeeded connecting to the engine

[lusty cape]

yes, it connects successfully but the pipeline never starts

its like something changed on dagger's end, maybe its what marcos is suggesting, we are not using the CLI, the pipeline is triggered programatically

i see this locally:

this is with latest dagger version...

[fossil spoke]

This means that the engine couldn't start

If you do a docker ps -a

You should see a stopped engine

And then you can do docker logs to see what's happening

[lusty cape]

i just re ran it and failed with the same error and the engine is still up

i can see the logs but they dont make sense to me unfortunately...

[fossil spoke]

I'll get home in 30 and we can check it together

[lusty cape]

thanks!!! ill be around

[fossil spoke]

ok, back. Have a few to jump into a quick #911305510882513037 session?

[lusty cape]

yep

[lusty cape]

thanks marcos for the help!!!

[fossil spoke]

we found the issue. Seems like gql was bumped from 3.5.3 to 4.0 according to v0.11.x's pyproject dependency def here https://github.com/dagger/dagger/blob/v0.11.7/sdk/python/pyproject.toml#L33 which caused the GraphQL introspection query to fail.

We fixed it by pinning the gql pacakge to its origina 3.5.3 version

this is important for future SDK releases so we validate that we always use the same major dependency versions 🙏

I'll open an issue in a bit so we don't lose track of it.

Thx @lusty cape for the patience and the inconvenience 🙏

[lusty cape]

thank you for taking the time