#Daggerize CI process with “external human” step

1 messages · Page 1 of 1 (latest)

radiant cedar
#

Consider a deployment or release workflow which is not fully automated, and perhaps cannot be since it involves something like legal compliance, internationalization QA, sending an email and waiting for a reply, or some other step where Dagger needs to stop executing for a while, some human somewhere needs to be put on task with inputs generated by the CI steps before it, that human generates outputs which are then fed back into the next CI step along with some unchanged outputs from the earlier steps, and then Dagger picks back up from here.

I’m curious about how to model this with any automation tool. I don’t have any existing examples, but I can think of a few things that would or would not work in theory.

For example, a naive (and unrealistic) solution would be to simply block a dagger function on user input. If the human task doesn’t take long and the human can be activated immediately to do the task, this is almost workable. But typically neither of these constraints will apply and then your CI is churning through n minutes of compute time waiting for the human to do their thing. On the bright side, your CI status (assuming timeout is impossible) will show the success or failure status of the entire flow, including the human process.

Conceptually, what might need to happen instead is the dagger process calls an external work management queue to activate the human step, uploads assets needed by that step (or perhaps even spins up servers on persistent infrastructure), serializes some of its own state to storage, then exits with success. Then when the human step is completed, a new Dagger process is started which receives input from the human step, recovers its state from storage, and continues. This does not sound super fun to model, although I could be overstating this. The ability to track the status of the full process from your SCM host requires it be able to enter a paused/pending state and then receive status updates far into the future.

woeful forum
#

Hello, In the first case, If i understand you correctly, you would have propritary functionality of your CI Platform being used for manual intervention / manual stages/steps and having user input. This would mean having several dagger function calls. This is what has been recommended to make benefit of say using CI native artifacts output or report features.

This would mean the dagger functino isn't blocked on user input, but blocked on the CI stage possibly?

Your second case very much reminds me of terraform state kind of. You have a process start, it completes... it sends off some kind of req to an external service for approval (maybe at this point, its even http timeout configured, waiting for a HTTP 200 back). This version seems a bit more complicated to lookafter though.

#

There was a very similiar situation/case asked about having user input into dagger and in the CI but its not possible because dagger cannot take user input from a CI environment from what I understand. I.e if dagger opened up a Terminal for user input, theres no way to get that tty working in the CI. So terminal() isn't suitable in the CI environment but is suitable for local

#

That case was more about 'are you sure you want to run this cmd? (yes/no) as a requirement.

#

the answer in that case is to utiilise the native CI features for manual intervention and then follow up with the dagger calls - But this isn't entirely like the case you describe i think

winter swift
#

Someone else recently asked about this and I proposed an idea using the native functionality of your existing CI provider.

#1277921913058099210 message

Are you using a CI provider already? If so, which one?

radiant cedar
#

@winter swift Took a look at that thread, thanks for the link. I've got some usage of GitHub Actions, GitLab CI/CD, and Bitbucket Pipelines under my belt. I'm most familiar with GitLab and Bitbucket, and while both have some way to do a manually-triggered deployment stage, I don't know that they offer a solution for injecting outside values mid-pipeline. (Bitbucket does have something called pipeline variables but they can only be used when manually triggering an entire pipeline from the start.)

dire oxide
#

@radiant cedar in a perfect world there would be a higher-level workflow system, built on top of individual Dagger pipelines connected by events. This is similar to what eg. Temporal provides. Legacy CI platforms bundle a workflow system, but it's too tightly coupled to everything else.

Eventually I think we may build that higher-level workflow system, and I expect others in the Dagger ecosystem will too (competition on higher-level products is a sign of a healthy ecosystem). In the meantime you can build your own, either by "hacking" it into your CI, or using generic workflow systems like Temporal, trigger.dev, etc

radiant cedar
#

As I was reading that other thread I was starting to come to the same conclusion -- I think I needed to slightly revise my mental model for where dagger fits into the CI picture, because in truth it really is lower level than a complete GH Actions/Bitbucket Pipelines/etc solution, and I definitely see value in keeping those concerns separate. I wasn't familir with temporal or trigger.dev -- will be giving those a look for sure

#

Thanks solomon!

dire oxide
#

The big picture is that we are unbundling CI

#

And we are doing so from the ground up - first the smallest possible component of a pipeline (the function); then the pipeline; the runner & orchestrator; artifact caching; the control plane; visualization; collaboration; and eventually: event routing and overall workflows