Monorepos with multiple modules

[pastel saddle]

Is dagger functions supposed to pick up functions from imported objects, like in Test and Lint this example, because it does not for me:

# src/my_module/main.py
import dagger

from .test import Test  # in src/my_module/test.py
from .lint import Lint  # in src/my_module/lint.py

@dagger.object_type
class MyModule:
    @dagger.function
    def test(self) -> Test:
        return Test()

    @dagger.function
    def lint(self) -> Lint:
        return Lint()

[pulsar wolf]

No, it only lists functions defined in the current module. The .help builtin in dagger shell lists both ( dependencies can be executed as command)

dagger -c .help

[pastel saddle]

How should we implement the monorepo pattern if the imports aren't visible?

[pulsar wolf]

Well the dependencies are visible to the shell and to the SDKs (so one module's code can call its dependencies)

I agree dagger functions should probably be made consistent with shell, but it's just a superficial UX issue in one command.

What aspect of the monorepo pattern do you feel is difficult right now?

[wooden cedar]

You have to follow the chain. The main object is just the starting point. So dagger functions test should give you the functions for the object that test returns (i.e., Test). Same for dagger functions lint.

[wooden cedar]

I think OP is asking about other types from his Python module, imported from other files, not dagger module dependencies. It's confusing but each file in Python is a "Python module". A group of modules is a "package".

[pastel saddle]

What do mean by they "are visible to the shell"? How would I call functions from imported modules? Could you point me to an example of your suggested monorepo organization? It would be even better if I could mix and match module languages in a monorepo. The documentation has no example (code).

[pulsar wolf]

For example the github.com/dagger/dagger module has a dependency called wolfi.

So you can do:

dagger -m github.com/dagger/dagger -c 'wolfi | container --packages=openssh,git | terminal'

The Dagger docs module has a dependency on the Dagger CLI module (to generate the CLI reference and include it in the docs build).

So you can do:

dagger -m github.com/dagger/dagger/docs -c 'dagger-cli | binary --platform=current | export ./dagger-cli'

To list dependencies you can call .deps in the context of any module, eg. dagger -m github.com/dagger/dagger -c .deps

[wooden cedar]

Yeah, there are two different concepts here. If you want to organize a single Dagger module that uses the Python SDK using multiple files, here's an example for you: https://github.com/dagger/dagger/tree/7187d42fd718909f91e5a06492a4485108e3c15c/sdk/python/dev/src/python_sdk_dev. It makes no difference when calling functions from this module via the CLI how its files are structured.

But since you mention you want cross language, that's done by creating multiple Dagger modules and dagger install as dependencies like Solomon mentions. One thing to understand is that you won't use Python's import system to use these types from other Dagger modules. There will be automatically generated client bindings through the dag client instance.

[pastel saddle]

True, I did mix it up 🙂 While looking at your link, I remembered another question: why do we duplicate the commit hash like so?

"source": "github.com/kpenfound/dagger-modules/dockerd@57352e06a1cfbcb5307c009d37c0201b2719b935",
      "pin": "57352e06a1cfbcb5307c009d37c0201b2719b935"

[wooden cedar]

The point of source is you can use a branch, then the pin will resolve to the commit at the time you dagger install or when you dagger update. So it doesn't usually look like that.

[pastel saddle]

How would it look like for a branch? If you can specify the commit in the 'source', why is 'pin' needed?

[tired island]

because if a user does dagger install github.com/foo/bar@main we snapshot the commit at that time and add it as a pin so they next time anyone does dagger develop, we take that exact same version

otherwise, we don't have an ability to always be reproducible

that's similar to the npm.lock and cargo.lock models

you can always run dagger update if you want to bump to the latest main afterwards

[pulsar wolf]

btw the commands I shared benefit from pinning. When you run eg. dagger -m github.com/dagger/dagger -c 'wolfi | container --packages=openssh,git | terminal' you get the exact version of the wolfi module installed in that version of the dagger module

[pastel saddle]

If I understand correctly the pattern is to create a master module that declares the others as dependencies? And doing so enables you to mix sdks?

[pulsar wolf]

Not necessarily. That's how we've done it, but the most important part is to create a module for each logical component in your monorepo, and make it individually useful for developers interacting with that component

While you do that, you can model dependencies between those components as dagger module dependencies

as part of that process, it's pretty typical that the top-level of the monorepo itself is a component, and therefore a natural place to have a module with functions of its own, and most probably dependencies on other modules in the monorepo. But it's not mandatory. And if you do have a top-level module, it doesn't necessarily need to have direct dependencies on all the other modules in the monorepo

[pastel saddle]

Thanks. I assume you do cycle detection.

[pulsar wolf]

Yes (I think!)

[tired island]

@pulsar wolf I thought we couldn't hit a cycle issue since the graphql API can be infinitely nested? Given that we register modules under its own nested scope?

[pulsar wolf]

Yeah good question, I don't actually know what happens if A imports B and B imports A. But I know that it's handled and hasn't been an issue for anyone that I remember.