#Upgrade dagger modules

1 messages · Page 1 of 1 (latest)

urban niche
#

Hi,
We are using dagger and now we begin to have multiples dagger modules across our repo (internal modules + end user modules).
I would like the have a dagger module in charge of the maintenance like upgrading dagger modules and some manifests.
For the dagger module I begin to do:

@object_type
class Maintenance:
    source: Annotated[dagger.Directory, Doc("Source directory"), DefaultPath("/"), Ignore(
        ["**/*.env", ".venv/"])]

    @function
    async def upgrade_dagger(
            self,
            version: Annotated[str, Doc("The version of dagger to upgrade")],
    ) -> dagger.Directory:
        repo_location = "/opt/repo"
        dagger_ctr = (
            dag.
            container().
            from_(f"registry.dagger.io/engine:v{version}").
            with_directory(repo_location, self.source)
        )

        for i in await self.source.glob("**/dagger.json"):
            dagger_ctr = (
                dagger_ctr.
                with_workdir(Path(f"{repo_location}/{i}").parent.absolute().__str__()).
                with_exec(["dagger", "develop"])
            )

        return dagger_ctr.directory(repo_location)

But it seems dagger develop need to be connected to the engine.
I would like to know if I can run something like dagger in dagger or connect my dagger engine to this function with a socket or something like that ?

past sand
urban niche
#

I will try that I forgot to check options from develop

urban niche
#

I just test it and it doesn't work correctly.
The command upgrade 3 modules but I have 23 modules.
I launch the command from the root of my repository, the command upgrade:

  • the root module
  • 2 modules from the local daggerverse of the repo but in this directory we have 6 modules
#

I think I just understand, the dagger develop -r is upgrade the current modules and dependencies from the dagger.json but I'm a kind of monorepo so we have several modules at different places and not all a linked they are using some common modules

past sand
# urban niche I think I just understand, the `dagger develop -r` is upgrade the current module...

I think I just understand, the dagger develop -r is upgrade the current modules and dependencies from the dagger.json but I'm a kind of monorepo so we have several modules at different places and not all a linked they are using some common module

yes, that's correct. Well.. I see two possible options from here:

  1. Create a "meta" module that has a dependency with all the modules in your mono repo and run dagger develop -r against that.

  2. Follow the approach you were describing initially of running scanning the monorepo for dagger.json files and then running dagger develop on each directory that you find

urban niche
#

yes at the end I did the option 2 with a shell script.
I was trying to have that in dagger but at the end it's maybe not a good idea

past sand
#

I was trying to have that in dagger but at the end it's maybe not a good idea

it's a very good idea actually. You were very close with your initial approach. In order to make dagger develop withing a with-exec you need to set the experimentalPrivilegedNesting flag in that call

urban niche
#

ok but one thing I was blocked I think or maybe the flag experimentalPrivilegedNesting unblock me, is the fact how my dagger develop from the container can access my engine ? (from the CI the dagger engine is setup over tcp socket or unix (don't remember) but from my laptop I think the dagger cli is using a container discover with container://...

past sand
urban niche
#

ok I will try that today or monday

urban niche
#

Hi I just tried to add the parameter it seems better but I have another issue:

Container.withExec(args: ["dagger", "develop"], experimentalPrivilegedNesting: true): Container! 5.6s
  ▶ dagger develop 4.6s

  6   : ┆ ┆ upload /opt/repo/airflow-dags from 0jnmpwbqb8zdbytadol2zrrfw (client id: qrksulpyv9t7a3iimmjkp6dw8, session id: 9yhzhte84dpo6673ql1xzmnzo) (include: .env)
  7   : ┆ ┆ filesync
  7   : ┆ ┆ filesync ERROR [0.0s]
  7   : ┆ ┆ ! failed to receive stat message: rpc error: code = NotFound desc = get full root path: rpc error: code = NotFound desc = eval symlinks: lstat /opt/repo: no such file or directory
  6   : ┆ ┆ upload /opt/repo/airflow-dags from 0jnmpwbqb8zdbytadol2zrrfw (client id: qrksulpyv9t7a3iimmjkp6dw8, session id: 9yhzhte84dpo6673ql1xzmnzo) (include: .env) ERROR [0.0s]
  6   : ┆ ┆ ! failed to get snapshot: failed to receive stat message: rpc error: code = NotFound desc = get full root path: rpc error: code = NotFound desc = eval symlinks: lstat /opt/repo: no such file or directory
  5   : ┆ Missing.directory ERROR [0.0s]
  5   : ┆ ! failed to get snapshot: failed to receive stat message: rpc error: code = NotFound desc = get full root path: rpc error: code = NotFound desc = eval symlinks: lstat /opt/repo: no such file or directory
  2   : moduleSource DONE [1.1s]

  8   : ModuleSource.localContextDirectoryPath: String!
  8   : ModuleSource.localContextDirectoryPath DONE [0.0s]

  9   : ModuleSource.sourceRootSubpath: String!
  9   : ModuleSource.sourceRootSubpath DONE [0.0s]

  3   : develop
  4   : ┆ develop . DONE [3.5s]
  3   : develop DONE [3.5s]
#

When I'm in the container I can find the repo:

dagger / $ ls opt/repo/
README.md           airflow-dags        applications        ci-tools            devops-tools        frontends           libraries           pvc-injector        scripts             untracked_files     versions.yml
__init__.py         airflow_kube_tasks  celery              dagger.json         flows               infra               manual              pyproject.toml      tools               uv.lock
dagger / $ cd /opt/repo/tools/daggerverse/python

The path in the container should match the path on the filesystem in this case ?

#

the function now looks like that:

    @function
    async def upgrade_dagger(
            self,
            version: Annotated[str, Doc("The version of dagger to upgrade")],
    ) -> dagger.Directory:
        repo_location = "/opt/repo"
        dagger_ctr = (
            dag.
            container().
            from_(f"registry.dagger.io/engine:v{version}").
            with_directory(repo_location, self.source)
        )

        for i in await self.source.glob("**/dagger.json"):
            if not "sdk/runtime" in i: #skip some dagger folder from codegen
                dagger_ctr = (
                    dagger_ctr.
                    with_workdir(Path(f"{repo_location}/{i}").parent.absolute().__str__()).
                    with_exec(["dagger", "develop"], experimental_privileged_nesting=True)
                )

        return dagger_ctr.directory(repo_location)
past sand
#

@urban niche does it also fail if you add a Terminal call right before the dagger develop exec and you manually cd into the directory and call dagger develop manually?

urban niche
#

yes it's working, I'm not sure to understand the diff

# 
        for i in await self.source.glob("**/dagger.json"):
            if not "sdk/runtime" in i: #skip some dagger folder from codegen
                dagger_ctr = (
                    dagger_ctr.
                    with_workdir(Path(f"{repo_location}/{i}").parent.absolute().__str__()).
                    terminal(experimental_privileged_nesting=True).
                    with_exec(["dagger", "develop"], experimental_privileged_nesting=True)
                )
past sand