My immediate use case for this: running | Dagger | Page 1

livid shale May 27, 2024, 1:30 PM

#

One step ahead of how pocketci works. We use Dagger in dagger through the SDK and the pocketci agent being the API that receives webhooks from github/gitlab and calls the repositories "Dispatcher" module. no yaml in sight.

Haven't sat down to figure out how to make it so that the agent itself runs inside dagger, as a module, instead of it being the top level binary. On the way!

spice jewel Jul 24, 2024, 5:14 PM

#

@livid shale

Necroing an old thread. I was kind of curious about this because I'm looking at doing something similar.

The arch I was looking at was

DaggerSession1 -> Run a Dagger that spawns your locally ran CI as a service (Let's call it my-ci). DaggerSession1 would bring my-ci online and then emulate a webhook call to my-ci that mocks an event from a platform (github, gitlab, jenkins, whatever) -> my-ci spawns DaggerSession2 to run the actual CI job, exactly as if a webhook had come in to a long running service

Is that also similar to what you are describing?

#

In this approach, the difference between remote and local is that DaggerSession1 would be emulating the remote environment, implying that on a shared CI where everyone is working off (non-local) DaggerSession1 would not be necessary, and instead the webhook would simply go straight to the long running non-daggerized service.

livid shale Jul 25, 2024, 3:09 PM

#

Hey @spice jewel!!!

In this setup, where does the matching of webhook -> dagger call happen? Meaning, how do you know which specific call to make based of a given webhook?

spice jewel Jul 25, 2024, 3:09 PM

#

In this case I was considering implementing a custom CIWebhookEventHandler as a dagger module

livid shale Jul 25, 2024, 3:10 PM

#

That handler would also be used in the remote service?

spice jewel Jul 25, 2024, 3:10 PM

#

which would be run locally in the case of local, but as a long running webhook receiver service in this theoretical setup

#

precisely

livid shale Jul 25, 2024, 3:17 PM

#

Excelent. In that case, @elfin egret recently demoed exactly that. In the demo, he had a dagger module for his project and:

Run dagger call serve to start a local API
Sent a curl to that API with the payload of a github webhook
Wrote a handler that handled the webhook and run the corresponding function

In the context of pocketci, this is similar and something can also be baked into it. The idea we are playing with at the moment is to transform pocketci into a collection of functions (or modules) that you can include into your module and that make it so that handling different webhooks from different providers is a breeze. Ideal experience is for you to be able to just implement an "interface" with methods such as OnPullRequest, OnIssueCreated, etc. We started off with a single method that is called Dispatch that recevies the entire webhook json via a file and a pocketci module that parses said webhook and returns typed structures that you can use to check the event from any programming language. For example, for the case of a pull request:

func (m *Module) Dispatch(ctx context.Context, payload *dagger.File) (*dagger.Container, error) {
  event := dag.Pocketci().Parse(payload)
  switch {
  case event.OnPullRequest():
    // do stuff...
  }
}

Still a PoC, what we are trying to understand is if the experience of handling this webhooks can be superior or as good to the experience of writing YAML for it (a github workflow basically)

spice jewel Jul 25, 2024, 5:30 PM

#

This is great, it seems we are thinking in the same space. Something like this

(https://mermaid.live/edit#pako:eNp9U99v2jAQ_lci9xVaSGCk0bSHQbWibWoETJPa7MEkB7EwduYfLQz43-fEIQklm19y9313vjt_lwOKeQIoQGuBs9RZTCLmmCP10gIhjzegxtM5iFcSg2XzM57-hGXK-ebhFZh6xCyhIGq6QEMsZBOcEJlhFac5ZlFgydmsSj7sFAiGafg0X9S5X4h61Ms786F4eTcmIqYwntb8Nx5jOtOMgfjOE03h3wUmeL0Gkc8kJeGsMRMlpuva_6wJTb6SEvlPxyz5YQadMtP4Cscw3Wa0vuWJhZrSGfzWINUlLNN3xcLZS4QKK0K_3nMmvJVdmHuLxNy4ZmzaFTeBjPJ9yVqn4htDNtVwut1PzrHU_di-G617USRerUQDKAKa61HwLc9qW5hn-I2VOjqljMd2Xa2mzsc8r9bTchfKOFVEOLN06RSEfeM6TaaN-ErHyq1zKu7s2VGrx7dkPbp9qtaFiphUewrN4BWhNLjp9T3X7XWkEnwDwY3neaXdfSOJSgM320UMddAWxBaTxPzth7xshFQKW4hQYMwEi02EInYycVgrPt-zGAVKaOggwfU6PTs6S7CCCcFm97coWGEqDZph9sz5hY-CA9qhwL2_v3X7A8_vD3u-2xu6gw7ao6Drf7j1ByN_2B_6I2_kDwanDvpTXNE__QWIe3Tb)

then? In this chart, the call to Parse is the EventParser, mapping webhook events to a CI-agnostic model (bring your event triggering software here, we will turn it into a CI call)

Is the code you are discussing publically visible? It is clear you guys have unwrapped this significantly more than I have but have a similar vision here (I was considering using a different underlying technology that would do something similar to PocketCI)

I think one fundamental question around the way I am envisioning it and the way PocketCI might work, is that once the event is received by the handler, are jobs required to be preregistered to PocketCI? or is PocketCI simply spawning jobs tied to version control in the deployed version of PocketCI e.g. a 1:1 of "Spawn Dagger Session" : dagger run

#

another way of phrasing this question is "how does PocketCI 'know' about possible CI jobs/configuration"

toxic turret Jul 25, 2024, 5:33 PM

#

I don't get the "spawn dagger session" part. At that point aren't you in a dagger session already?

spice jewel Jul 25, 2024, 5:34 PM

#

@toxic turret maybe I can update the chart. When running locally, everything before "Spawn Dagger Session" would already be happening in a locally run dagger session, and it would be Dagger in Dagger. But remotely, PocketCIService might be a long running server/service that runs outside of dagger, so it would not be Dagger in Dagger, it would be a regular hosted service spawning Dagger sessions for each incoming CI event

#

the implication here is that this would be in lieu of (in replacement to) github actions runner, kind of like how Actions Runner Controller (the helm chart) receieves github actions webhooks and spawns jobs based on the webhooks - in that case Actions Runner Controller has its own webhooks receiver that does something similar to what is being discussed here. The idea would be somewhat analagous to daggerizing ARC locally but using hosted, nondaggerized ARC when pushing to your remote VCS

#

(hopefully this makes sense)

toxic turret Jul 25, 2024, 5:40 PM

#

I see, the bottom box "DaggerCISession" doesn't have user-defined code in it, that's all in the middle box labeled "EndUserInterfaceImpl"?

livid shale Jul 25, 2024, 5:53 PM

#

I think one fundamental question around the way I am envisioning it and the way PocketCI might work, is that once the event is received by the handler, are jobs required to be preregistered to PocketCI? or is PocketCI simply spawning jobs tied to version control in the deployed version of PocketCI e.g. a 1:1 of "Spawn Dagger Session" : dagger run
In the current PoC, pocketci acts as an agent that is deployable and configurable. You specify which repositories it is in charge of handling and it makes the assumption that the Dispatcher function will be a part of the top level repository module. This works well with monorepos. So the setup would be: you deploy pocketci with your repo configured, you can also configure secrets here; you point your repo's webhooks to pocketci; upon receiving the webhook, pocketci interprets the event and the first thing it does is make a git clone at the corresponding SHA, computes some git metadata to enrich the context, then creates a container with this directory mounted and the dagger CLI installed and launches a dagger call to your module's Dispatch function. Then, within your module you can do pretty much whatever you want. Using the pocketci module is optional, if you want parse the webhook yourself you can

The nice thing about this is that pocketci can add additional metadata to the container where the Disptacher function will execute. You can have information such as the list of files that changed more readily available, and you can integrate it with secret providers more easily

#

Another thing I was playing with is seeing how well dagger views integrates with this. You can define views related to a given application in your repo, or subset of services within your application, and relate those views to differnet dagger dispatches so that only the code for those services will get executed. The way I match which dagger view will be active is using the list of files that were changed

#

You parse them on pocketci's side and you can check whcih one is active on the module's side. This can be done in many different ways, not sure if this one is the right one, the DX is not particularly great

toxic turret Jul 25, 2024, 6:01 PM

#

Note that views will soon be deprecated in favor of context dir

spice jewel Jul 25, 2024, 6:03 PM

#

Thanks for this, I'm digesting since there's a lot going on here. When we say "top level repository module" we mean something like ci or what you guys I believe call develop right? The entrypoint that contains the cli commands to call other modules?

livid shale Jul 25, 2024, 6:03 PM

#

Good to know. Not a problem though. I was just exploring a way to support the feature github workflows has were you only run your pipelines based on a given set of files that were changed. I used dagger views because they already had a list of globs and I could conceptualize an application with a list of globs

livid shale Jul 25, 2024, 6:03 PM

#

spice jewel Thanks for this, I'm digesting since there's a lot going on here. When we say "t...

Want to jump on dev-audio for a quick overview?

spice jewel Jul 25, 2024, 6:04 PM

#

I do, give me a sec

toxic turret Jul 25, 2024, 6:05 PM

#

I won't join this call even though I love this topic as you know @livid shale 🙂 I will only say that the final form of PocketCI in my view will involve zero yaml

livid shale Jul 25, 2024, 6:06 PM

#

toxic turret I won't join this call even though I love this topic as you know <@6283920878800...

We are on the same page there. I think the most interesting bit to explore is the check interface you mentioned in some channel (I'll look it up later for reference). Interfaces might be a great way of making everything pluggable

toxic turret Jul 25, 2024, 6:07 PM

#

Yes interfaces are the key 100%

livid shale Jul 25, 2024, 6:10 PM

#

At the moment pocketci relies on loose interfaces. There is no explicit interface to call to, but you have to implement the function with a given signature, otherwise the call will just fail

toxic turret Jul 25, 2024, 6:15 PM

#

I think as a simple start you could define 2 interfaces:

EventHandler with a generic handle() function for the raw event.
GithubEventHandler with more Github-specific functions like onPush etc

Then you provide an implementation of EventHandler that does the github-specific parsing and dispatches to user-supplied GithubEventHandler.

The result is 2 layers, one to build github-specific workflows, and one to build integrations with other VCS

#

Higher-level stuff like Checks etc can be experiments on top of all that

#

If you do the above 👆 you unlock 90% of the code you implemented in pocketCI (github parsing etc) and instantly it becomes accessible to the entire Dagger ecosystem to experiment with.

#

There's probably a 3d layer in there, which would be an interface specific to the individual git commit. SO MUCH of this over-complicated event processing mess that we call CI, boils down to running functions against a git commit.

#My immediate use case for this: running