matrix runs

[thorny laurel]

When porting a github actions matrix run over to dagger.

What's the recommended approach here?

Specifically for python.

Should I invoke a parallel executor for this so they all happen at once?

Thanks 🙏

[orchid timber]

You could do it by running multiple dagger calls, each of the matrix jobs will be on their own runner. You could also fan out inside the dagger code, and this will be limited by the host cpu/memory mostly since all of your tasks are fighting for the same VM resources.

[lime wasp]

I generally setup this in a way where I have a wrapper Dagger function that runs everything concurrently using the native language concurrency features and I also leave the door open to call these functions independently so I can still use matrices in my CI compute platform 🙏

[thorny laurel]

I found one of Helder's example - https://github.com/helderco/dagger-examples/blob/main/test_multiple.py

[thorny laurel]

What's your opinion on Helder's example? is there another example or different way to solve it

[lime wasp]

yes, Helder's example seems ok to me as a way to solve. As I mentioned above, I personally like structuring the code in a way that I can also call the functions concurrently via multiple dagger calls so I can use my CI native concurrent capabilities if needed

[thorny laurel]

@lime wasp can you invoke a "dagger call" via code? Rather than CLI.

The idea is to migrate away from github actions YML completely and let dagger take over

Such as "dagger call test_matrix"

[lime wasp]

what's your plan on getting rid of the YAML entirely? How are you planning to achieve that?

[thorny laurel]

@lime wasp not entirely. You still gotta invoke dagger engine from the YML.

But after that it's dagger o'clock

Are you suggesting to leave all the matrix stuff inside the YML file. And do dagger call inside the github matrix

[lime wasp]

no and yes. It's what I suggested above about having the ability to still use the matrix if needed. There are pros/cons about keep using the matrix.

What I'd initially do is to remove it and see how acceptable the performance is. Keep in mind that when removing the matrix, all your pipeline will run within the same VM which in some cases, might yield to slower executions depending on the resource consumption of the pipeline.

[thorny laurel]

@lime wasp yea, I got it.

[leaden cipher]

Yes, dagger call just builds a GraphQL query in the end.

Like Marcos is saying, I can choose to test the Python Dagger SDK with all supported Python versions concurrently: https://github.com/dagger/dagger/blob/a2d4b1f0cd11a0affde61836b980f4462174e92e/sdk/python/dev/src/main/__init__.py#L229-L232

Or just a single one: https://github.com/dagger/dagger/blob/a2d4b1f0cd11a0affde61836b980f4462174e92e/sdk/python/dev/src/main/__init__.py#L219-L225

Note that test_versions calls out to test(version) in a loop.

[fickle sage]

@leaden cipher @lime wasp do you think we should update our cookbook in python to take a different approach?

I have an example that I worked with Helder before that uses anyio and task groups

https://docs.dagger.io/cookbook/#perform-a-matrix-build

[leaden cipher]

Depends on the use case. For that cookbook, it's actually the better approach. Since it's building multiple files, just add them lazily to a directory and export the directory in the end. But I think it's good to create a new cookbook entry that uses concurrency, for example to test multiple versions. The build matrix example is building a go app in all SDKs, the new test matrix could test multiple Python versions in all SDKs 🙂

[fickle sage]

Great thanks!

I am thinking more of a brand new person coming to dagger and searching for matrix builds in our docs

I think the most common use case for matrix builds in traditional CI is multiple versions of a language/service

[leaden cipher]

Can be matrix "builds", matrix "tests", matrix <etc>. Depends on what you're doing, but it's not always a "build":

• https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/running-variations-of-jobs-in-a-workflow
• https://hatch.pypa.io/1.12/config/environment/advanced/#matrix