I've been iterating on a few ideas for

[past dew]

The current status of the project is very much a PoC/toy project. There are many things that are not implemented because I was focusing mostly on experimenting with the DX, testing what it felt like. A few of those limitations that are worth mentioning:

1. How should I integrate pocketci into my repo? At the moment, the only way to do it is to define the dispatching function (Dispatch for the lowest of levels, OnGithub and so on for higher levels) in your module. This presents two problems: i) modules with constructors that have custom parameters are not supported; ii) only one module per repository is supported. In many cases this is very limiting (e.g in the case of dagger/dagger).
2. Which events are supported?: Only GitHub pull_request & push are supported at this very moment
3. What about custom compute?: Everything is built into a single binary, no separation of orchestrating and compute logic yet (who receives the event & who actually runs the dagger call).

In the next few days I'm going to be fixing most of them (with a PoC mindset still) but I'm very interested in the thought and feedback of the community:

1. Having the ability to support multiple modules per github event is a must, mostly for monorepos. On top of that, I plan to add another convention: functions can have a Prefix and pocketci will call every function that ends with the relevant string instead. This means you can do things such as: LintGoOnGithubPullRequest and LintElixirOnGithubPullRequest within a single module, pocketci will call both of those functions upon receiving a github pull request. For modules with constructors, a very hacky solution is already tested: add a module-entrypoint to pocketci.yaml with the literal string that should be sent to a dagger call command 😆.
2. Adding support for more events at the moment is very involved, mostly because the code is a mess. I'm going to clean things up a bit so that adding events becomes just a few lines of code
3. As for custom compute, the concept we are playing with at the moment is making functions "self contained". Could a function, through its own parameters and metadata, define not only what should trigger it but also where it should run? It would be kind of cool to be able to do something like:

// +engine="<engine name/url/etc>"
func (m *Ci) LintOnGithubPullRequest()

At the moment this isn't possible. However, I was playing with a different idea that could feed into this. If we want a trigger that is: only call this function when a tag of the following regex gets matched, we would need to do some fancy stuff. What I ended up doing is requesting function developers to add a parameter that is called filter and add a default value to that filter that has the regex in question. I would then read that default value, interpret it and then make the dagger call with the real value only if the regex matched. This is definitely a bit hacky, but it keeps as close to the idea of "self contained" functions. Basically, something like:

func (m *Ci) PublishEngineOnTag(ctx context.Context
    // +default="v**"
    filter string,
    // +optional
    // +default="tcp://some_ip:8080"
    engine string) ...

Again, this isn't great, but it lets us play with this concept and see if it actually fits.

All of this is experimental and its mostly a thought excercise. But maybe we can take something good out of it that improves the experience of writing and maintaining CI infrastructure.

[olive perch]

1. As you know I love this 🙂

2. I want to find a way to bridge my Github Actions config generator to Pocket CI. I think of the generator as a "gateway drug" to pocketci. It introduces you to the concept of defining your triggers as code using Dagger. But it doesn't cross the line of running the triggers in Dagger: instead it generates the config for GHA to run it. The APIs are clearly becoming more similar, but there's still a gap. Can we bridge that gap? How? Should we?

One gap in the GHA generator is dynamic generation. GHA doesn't support continuation, so a GHA "workflow" can't just dynamically generate and register more "workflows". Maybe filling that gap can be the excuse to "sneak in" pocketci?

[past dew]

I need to play a bit more with GHA and see more closely how you are integrating with it. My first thought would be to match 1-1 the generation of a workflow with a given dagger call we would like to run in isolation. Lets take SDK as an example. We would generate one file for each of the languages. This would match having separate SdkGoOnPullRequest handlers on pocketci. One of the downsides is that retriggering all SDK pipelines would mean having to go intl each workflow and click tje rerun. Observsbility on dagger cloud would remain similar

[austere briar]

GHA doesn't support continuation, so a GHA "workflow" can't just dynamically generate and register more "workflows"

Actually.... no you, can't generate more workflows, but you can generate additional jobs and use the output of another job as the definition.

In other words, you still need a static workflow file, but you can generate jobs within that.

It doesn't necessarily solve the problem you are trying to solve.

[olive perch]

mmm

It doesn't directly solve the problem (because I'm constrained to the event triggers statically defined in the workflow), but it gives me one more tool to think of hacks

for example it makes the "mega-init workflow" idea slightly more doable, because that one workflow could then spawn more jobs

But it boils down to event routing. someone has to route events to dagger calls.

• With manual configs, the GHA control plane does the routing
• With "simple" generated configs, the GHA control plane still does that
• With a mega-init workflow that catches all events, it can be either 1) the workflow itself (can GHA routing be implemented in the workflow DSL?) or 2) each job (shell script wrapper, or dagger function wrapper). but then every job is scheduled on every event, impossibly inefficient
• That leaves pocketci which of course reimplements routing in dagger itself

[winged shard]

🤔 wait what 👀
how do you do this?

[austere briar]

I can't recall exactly. I've seen something like this by using the output of a previous job and unpacking it with fromJSON.

[umbral roost]

IIRC this is how you do it. https://docs.github.com/en/actions/writing-workflows/choosing-what-your-workflow-does/running-variations-of-jobs-in-a-workflow#example-using-an-output-to-define-two-matrices

cc @winged shard

[past dew]

Its interesting to think why these kinds of functionalities exist in common CI platforms.

Why generate different jobs based on a list of inputs when you can do that natively within your programming language? In my experience it boils down to user control: if I have a list of jobs, each job will have its own check. I will be able to see its status, logs and re-trigger it individually if it fails. I can control the internal steps of a DAG and only focus on the parts I care about.

I've been thinking if doing something like that would be possible in the context of pocketci. If we take a very simple example of a pipeline that runs both Lint and Test on a pull request:

func (m *Ci) OnGithubPullRequest(ctx context.Context,
    // +optional
    // +default="**/**.go,go.*"
    onChanges string,
    // +default="synchronize,opened,reopened"
    filter string,
    src *dagger.Directory,
    eventTrigger *dagger.File,
    ghUsername, ghPassword *dagger.Secret) error {

    var g errgroup.Group
    g.Go(func() error {
        _, err := m.Test(ctx, src, ghUsername, ghPassword).Stdout(ctx)
        return err
    })
    g.Go(func() error {
        _, err := m.Lint(ctx, src).Stdout(ctx)
        return err
    })
    return g.Wait()
}

This function gives me a single check. Observability for both steps is mixed in a single one and if we had the capability of re-running implemented we could only re-run the entire step. I could instead write two separate functions that get triggered on the same event, and for this small example it might be better. But I can see others where that wouldn't be possible (particularly matrix-based examples, a recent one we did with @umbral roost comes to mind: https://www.youtube.com/watch?v=mPO9zVjGquo).

Caching only partially solves the re-running part. It only solves it if the user implemented it "correctly". Meaning, they don't have any manual or accidental cache busting in their pipeline. And we are still treating the execution as a global pipeline, rather than extracting the parts of it that matter. The tricky part is defining what exactly "matters". Is every top level funcion call a separate step? Should it be user controlled? I know we are deprecating the Pipeline function, but if exposed that could give users control to create different parts of their pipeline that could potentially be extracted into separate steps that can be re-run. I'll play around a bit with the telemetry data we expose and see if I can at least have a visualization PoC

[umbral roost]

Why generate different jobs based on a list of inputs when you can do that natively within your programming language? In my experience it boils down to user control: if I have a list of jobs, each job will have its own check. I will be able to see its status, logs and re-trigger it individually if it fails. I can control the internal steps of a DAG and only focus on the parts I care about.

I guess one valid use-case for having the job generation like Github does it is the ease of spinning new jobs which could potentially be executed in different compute. Having said that, I agree that it's way easier to express by code than doing this ugly matrix dancing.

On the other hand, the fact like they did it how they did it is quite clever as it allows for job re-running quite easily since the inputs for each job are properly serialized.

1/2