#Running a CI Matrix in parallel

1 messages · Page 1 of 1 (latest)

azure sky
#

@novel sequoia Is there a Dagger-specific solution to this? For any SDK?

Or would it need to be split into separate jobs via YAML workflows?

How would this be achieved locally?

Context

Adrien has this function:

    #[DaggerFunction]
    public function testsMatrix(): array 
    {
        $result = [];

        $result = [];
        foreach (['8.2', '8.3', '8.4'] as $phpVersion) {
            foreach (['6.4.*', '7.2.*', '7.3.*'] as $symfonyVersion) {
                $result[] = $this->test($source, $phpVersion, $symfonyVersion)->all();
            }
        }

        return array_merge(...$result);
    }

Where $this->test(...)->all() runs CI for one version.

brave coral
#

AT the moment I fixed it with YAML in github actions. With matrix jobs. But locally it can't run in paralell ATM.

novel sequoia
#

🤔 so, you can parallelize it for sure

#

not sure how you do it in php, but for example, in go you'd use a go statement, or in typescript you'd use async/await

#

in terms of output as you-go 🤔 i think there needs to be some php otel support? i'm not sure if that's there

#

essentially this

azure sky
# novel sequoia in terms of output as you-go 🤔 i think there needs to be some php otel support?...

I think some work for OTEL support was implemented for PHP.
https://github.com/dagger/dagger/issues/8088

But not enough to get as-you-go output (yet).

GitHub
PHP: Implement OpenTelemetry in runtime · Issue #8088 · dagger/d...

Make use of OpenTelemetry in PHP SDK runtime so we can get real-time feedback as opposed receiving all output only once complete. Show step details in TUI Implement Otel within SDK Use custom spans...

novel sequoia
#

yeah, so, that's what's missing - there needs to be some live span support for there to be live output - otherwise, you just get the full output when a thing is completed

azure sky
#

Right okay, I think I'll add running test matrices locally as a use-case in the issue.

brave coral
#

thank you for your inputs. Natively there are several approaches to async in PHP. You have whole frameworks (like swoole / reactphp), process forking, etc etc. What would be the best way to handle this ? simply requiring it ?

azure sky
brave coral
#

What do you mean ?

azure sky
#

So, Dagger cares about two things:

  • The arguments
  • The return value
#[DaggerFunction]
public function (
    // dagger cares what goes in here
): Thing { // dagger cares what you return
    // Dagger does not care

    // Still doesn't care...

    // Still not caring...

    return $thing; // Dagger cares
}

So you can do this:

#[DaggerFunction]
public function(string $myArg): string
{
    $coolClass = new CoolClassFromPackage();
 
    $coolString = $coolClass->doCoolThing($myArg);

    return $coolString;
}
#

So, you can do a composer install inside your module, like normal

#

You can use the package to do whatever you want inside the function body

#

But the package isn't exposed to Dagger. So Dagger can't pass it around.

# 
#[DaggerFunction]
public function(CoolClassFromPackage $coolClass): string
{
    // do cool things
}

That will fail because Dagger can't receive an object it doesn't know about

# 
#[DaggerFunction]
public function(): CoolClassFromPackage
{
    // do cool things
}

That will fail because Dagger can't return an object it doesn't know about

# 
#[DaggerFunction]
public function(): string
{
    $coolClass = new CoolClassFromPackage();

    // do cool things
}

That will work because Dagger doesn't do anything with it, it's just normal PHP

brave coral
#

yes Ok. Fully understand this part 😉 just curious to be sure I do not mistake things. How does dagger handle the multi-lingual support ? Is it AST converted to graphql ? Or is it running multiple language internally to make it work ?

azure sky
#

cc @novel sequoia in case I'm mistaken. 🙏

My understanding is Dagger deals in GQL.
Each SDK translates GQL into their respective language (and back).

Lets say you call dagger call test --php-version=8.3 --symfony-version=7.4

  1. Dagger sends the GQL query to the PHP SDK
  2. The PHP SDK will
    a. Receive the call from Dagger as GQL
    b. Translate php-version and symfony-version into PHP
    c. Execute the PHP code inside the test function
    d. Translate the return value back to GQL and send it back.
#

so inside step 2.c. there is no GQL, no other language is playing a part, it's just PHP

novel sequoia
#

graphql is the layer that everything connects to the engine through

brave coral
#

yes but how does it work with external modules ? Let's say I require a phpstan dagger module which is written in go. How does it work ? It would need to "invoke" the phpstan module in go itself am I wrong ?

azure sky
#

You are correct, if you use a module written in Go. Then it will be executed in Go.

  1. You install a PHPStan module written in Go.
  2. This gets added in with the rest of Dagger's GraphQL schema
  3. The PHP SDK regenerates your generated directory.
    a. If you dig into it at this point, you'll find some generated PHP code for the phpstan module.
    b. If you look at the code, it's all basically wrappers around GraphQL queries.

Then you run this function

#[DaggerFunction]
public function(Container $container): Container
{
    return dag()->phpstan($container)->analyse();
}

When use dag() you are basically just making a fancy GraphQL query, that sends Dagger off to get the result from the Go module.

novel sequoia
#

both of them speak graphql - the api is generated (to speak graphql on each side). you just call a function, and underneath the hood it's making graphql calls

#

the engine brokers everything

brave coral
#

yup okay. So then it executes go code then. And if I write something in PHP and someone else requires it, then my PHP code will be executed. Nice. Because was'nt totally sure how it worked. Wondered if it was using AST to converti it fully to GQL which would have been a nightmare to create and limited functionnality I guess. But that's all good. Meaning we could offload the async / await to other languages.

azure sky
#

You could yeah.

#

I can imagine a scenario where someone might have a PHP back-end with a JS front-end:

I've not tried it, but they might want a PHP module for the back-end, and a Typescript module for the front-end. Then run them using whichever supported language has the best support for coroutines

#

That example might be contrived or over-engineered 🤷 but it's entirely possible

brave coral
#

yeah just trying to think through what is and is not possible.

frank lake
brave coral
#

not really... Fibers allows to suspend and resume whenever there is no activity in a callback for example. So it allows that when your code is waiting for something (http client, sql query, else...) instead of waiting it can give back control to execute something else and get back to it. But in the case of dagger it is always doing something... I think the best approach would be to use pcntl fork of prcess

brave coral
frank lake
brave coral
#

yes but the PHP engine itself will see it as "working" (I think) so not sure Fibers applies here. Sadly pcntl is not available on the PHP runtime. I'll try to submit a PR to update PHP to 8.4 (is there a polivy about such upgrades ?) and see if we can add pcntl (i'll try locally first of course)

frank lake
#

I think; from the examples I've seen, the trick is to use a tick function to manage suspending and resuming the fibres

frank lake
#

oh, seems that no longer works; was patched out in 8.3

azure sky
#

So in this instance, if upgrading to PHP 8.4 allows for concurrency in your CI. That would be a pretty good reason. 😄

brave coral
#

not really... but upgrading php to 8.4 is also a way to allow contributors to execute their code with the latest syntax. Shouldn't be a way to register each module to say on which version it should run in (all sdk concerned)

azure sky
#

I'm not sure if other SDKs allow specified language versions, but I assume it would be quite the feat to implement.

#

PHP 8.4 does offer a few nice array & multi-byte string functions. So if it doesn't break anything, I'm not opposed. cc @frank lake ?