Hi all, was searching for a bit but

[hollow comet]

When you execute dagger run you can pass in an environment variable that allows you to execute on a different dagger engine environment, https://docs.dagger.io/541047/alternative-runtimes/#run-the-dagger-engine-container-manually

The simplest way to approach this in my mind would be to have multiple calls to dagger run where some run locally and some use this env var.

There could be some complexity if you need to pass some state from one step to another but would be interesting to work through that together.

[tender walrus]

yes, you can totally do this. What's your current setup @runic jolt ?

[runic jolt]

Hi guys, thanks for your answers. I have a really simple setup: currently we're completely on-prem. We use neither Kubernetes nor Docker swarm, just good ol' dockerfiles and docker compose. We have several runners for building+testing, most on the same server. The ones running on the same server are interchangeable and were only deployed to parallelize, so can be replaced with a single Dagger engine. However, we have several jobs that need to be run on specific machines: performance test and deploy jobs (need to restart services). Ideally, I'd be able to move content freely across those machines (just as I can move content across host and containers, as shown in the dagger demo). However, a less ideal but still nice solution would be to only move images across hosts, using the docker registry. Key issue to me is to be able to control which jobs get executed and how they are executed (i.e. the flow of the pipeline) entirely from the Go code. I would then have just one Gitlab job - one that runs dagger pipeline. My current (very shallow) understanding of Dagger only allows me to run a pipeline locally, and if a given function has to be run on a remote machine, I have to have another Gitlab runner, which starts another Dagger pipeline at that specific machine. This means I again need to go back to controlling how and when this remote job starts using Gitlab YAML instead of Go code - and this is what I'm trying to avoid in the first place. Is this something that Zenith could help me with? Deploying specific Dagger function (deploy/performance test) on those remote hosts? Anyway these are my thoughts - a complete Dagger noob, let me know what you think and if you have some related docs I'll be happy to read!

[tender walrus]

thx for describing your setup! In this case I'd recommend what @hollow comet was recommending above with the addition of using Dagger Cloud as a cache orchestration layer across your different hosts. With this setup, you could have a single Gitlab CI workflow which does something like

_EXPERIMENTAL_DAGGER_RUNNER_HOST=tcp://hosta dagger run build
_EXPERIMENTAL_DAGGER_RUNNER_HOST=tcp://hostb dagger run test

^ in that case, build can run in host A and test can run in host B.

The reason why I'm suggesting using Dagger Cloud is so caching can be efficiently re-used across different hosts. You could avoid using Dagger Cloud and setup an intermediate registry as you mentioned but we generally don't recommend doing this since caching is slower, inefficient and has a lot of caveats when using a stand-alone registry. If you decide to go this route, you'll find yourself spending a lot of time re-architecting your pipeline to accommodate all the registry limitations.

[runic jolt]

Hi Marco, thanks for your answers. I'm gonna experiment some more to understand whether running those two commands sequentially gives me what I need.

One small remark is that Dagger Cloud is probably a no-go for now. The costs would probably be quite low but our management is non-technical finance people - they're very conservative and prefer to keep everything on-prem. I think we'll slowly get to cloud but we're not there yet.

[tender walrus]

One small remark is that Dagger Cloud is probably a no-go for now. The costs would probably be quite low but our management is non-technical finance people - they're very conservative and prefer to keep everything on-prem. I think we'll slowly get to cloud but we're not there yet.

we're developing a solution for some partner customers so they can store the data in their own infrastructure if that's highly required. Depending on your size and volume, we might be able to offer this to you if that's something that might allow you to move forward. cc @hollow comet

[runic jolt]

Thanks for offering this! I'll go back to you when we have a better PoC ready and convince the stakeholders to reserve time to move away from YAML-based CI/CD that we run currently. This might take some time though since we don't have a dedicated devops, so it is entirely a side project for now.

[runic jolt]

Hi guys, just FYI if someone has a similar problem in the future, I found a solution that fully fits our needs:

• have a separate GitLab job that can only be triggered via API call
• this job should do nothing more than start a container based on an image built and pushed to your working registry from Dagger pipeline.
• trigger the job via an API call from Dagger, if the pipeline code decides it is necessary.
  by appropriately passing env variables in the API call the triggered job appears as a "child" of the dagger job, so also all acceptance criteria work (e.g. this and that job needs to pass to proceed)