#What is the common thinking with regard
grim wind
I can see two options:
- Polling: my code is frequently checking the S3 bucket for videos it hasn't seen before.
- Events: when a new video was uploaded, S3 sends an event to my code, which then launches the pipeline.
Besides not being very efficient, polling also has the problem that I need to re-implement it for each type of backend that should kick off another pipeline run, plus all the state management about which videos it has already seen.
Events would be elegant, but not all backends support it (e.g. SQS or emitting a webhook request) and I would have to implement that myself, too (potentially for each pipeline).
Concourse uses polling, and it has the idea of a resource, which encapsulates all the logic of dealing with a backend. When something interesting happens, the resource emits a new version, and the pipeline starts. In addition, Concourse also established a respectable ecosystem of resource types¹, allowing me to interface with e.g. S3 or GitHub releases with just a few lines of YAML.
Is the triggering of a pipeline something people have thoughts or opinions about, or is that outside the scope of what Dagger is trying to accomplish? Should I better use something like Argo?
tight hornet
The current idea is to combine Dagger with an external system that handles triggers. Eventually we may add triggers to Dagger but not in the short term.
However, Dagger will get you pretty far without an intricate event system, or fancy resource system. For example you could run your S3-download pipeline in a cron job, and just rely on Dagger’s cache.