gitlab

[rare heron]

The most obvious way to use Dagger with Gitlab is to run Dagger pipelines from a Gitlab CI job. Here's a reference on how to do that: https://docs.dagger.io/7442989/cookbook/#gitlab-ci

There is also a guide on using Dagger with GitLab CI/CD and Google Cloud: https://docs.dagger.io/759201/gitlab-google-cloud/

[candid basin]

I have so many noob questions today, sorry. 🙂 Is it possible to use Dagger with GitLab using buildkitd directly, foregoing DinD and the docker CLI? We have an existing buildkitd cluster that is working quite well with Istio circuit breaking and horizontal scaling, and I think for the moment we'd like to keep that.

[rare heron]

@candid basin you can't use your existing buildkit deployment; but you can deploy the dagger engine in a similar way, and bypass dind as a result. We have docs for that.

The reason it won't work on a vanilla bk is that we wrap buildkit pretty heavily for some of the features to work, for example service containers that rely on a runc shim for networking

Also, we can't reliably support a full dagger / buildkit version matrix

[candid basin]

Oh very interesting.

yep, makes sense. Thanks!

[rare heron]

Note, we're very interested in learning more from your buildkit setup, it seems more advanced than 99% of what we see out there; if it works well for you I want to make sure you can do the same thing with Dagger, and ideally share it with the rest of the community 🙂

[candid basin]

Oh, sure thing! I'd love to share. It's not that advanced 🙂 but we run a fork with some minor tweaks to enforce our own gateway frontend, and another patch to isolate the auth token cache space for registry pushes. We try to work with upstreams whenever possible, and the latter might be merged in the near future hopefully. Other than that, it's just Istio enforcing circuit-breaking on max connections, and a change to buildctl (buildctl --wait) that is now in v0.12 that has the client perform a warm-up request with slightly different gRPC options so it gracefully handles the circuit breaking

our terraform is here https://gitlab.wikimedia.org/repos/releng/gitlab-cloud-runner/-/tree/main/buildkitd

and our fork of buildkit lives here https://gitlab.wikimedia.org/repos/releng/buildkit

(still need to do that rebase on v0.12)

[rare heron]

is your buildkit cluster separate from the Gitlab CI cluster, or do they coexist on the same cluster?

[candid basin]

For now, our GL runners and buildkitds are on the same kubernetes cluster, same namespace, but there's no functional requirement for them to be. However, for performance reasons it's important that they're on the same network. We also have a homegrown registry there for cache exports/imports between jobs, temporary image storage for a build-once-run-many pattern in .gitlab-ci.yml files.

Our registry is written in Python and does reference counting for more aggressive GC. It's here https://gitlab.wikimedia.org/repos/releng/reggie

We also have a "trusted" runner space on production hosts that is set up quite differently. It's there that we allow pushes to our registry using the GitLab JWT, deb package publishing, etc.

[rare heron]

Super useful, thanks

Here's an outline of the current state of the art for an equivalent Dagger deployment:

• To avoid dind (among other issues), you would add a DaemonSet to the same kubernetes cluster where you run Gitlab CI runners. We actually recommend sharing the same cluster. DaemonSet ensures maximum reuse: essentially each node on the cluster gets a dedicated "pipeline coprocessor" that pods on that machine can handoff Dagger workloads to. Here's a guide with more details: https://docs.dagger.io/194031/kubernetes

• There is no Dagger-specific load-balancing, all jobs originate from a pod on the same cluster (specifically a Gitlab CI runner pod), and they always get routed to the engine running on the same kub node as the client pod. Effectively, Dagger piggy-packs on your CI infrastructure control plane for dispatching compute. This usually makes life easier for teams that have already invested in their CI infrastructure. One less moving part to worry about.

• We do expose the underlying buildkit cache-export features, but recommend against using them. There are fundamental limitations to the cache-export design, which become more and more apparent as you scale. Perhaps you have already encountered some of them 🙂 Instead we recommend using a centralized control plane which can collect telemetry from all engines, and use that to intelligently orchestrate the movement of cache data between each engine's local filesystem and a centralized storage service, as well as between engines. This should happen continuously in the background, instead of being a blocking operation at the beginning and end of each call. It should also persist cache volumes (those are not included in bk cache export). Anyway, we implemented all the above in a commercial control plane we call Dagger Cloud, it's available in Early Access as of this week, if you want to give it a shot 🙂 https://dagger.io/cloud

(There are additional nuances but that's the gist of it)

[candid basin]

centralized control plane which can collect telemetry from all engines, and use that to intelligently orchestrate the movement of cache data between each engine's local filesystem and a centralized storage service, as well as between engines. This should happen continuously in the background, instead of being a blocking operation at the beginning and end of each call

That's really smart! Yes, I think the biggest limitation to cache exports that we've run into so far is the image export operation itself. It's quite slow.

That and it's difficult to reason about which refs are (probably) going to give you the most relevant layers. We've tried to abstract some of that for people with our small wrapper client, Kokkuri https://gitlab.wikimedia.org/repos/releng/kokkuri

I'm going to see about doing an evaluation of Dagger on our CI cluster. As you can tell, we have a lot of homegrown infrastructure here and I'd love to get away from it. We just haven't seen anything that potentially fits all of our requirements until now. 🙂

[rare heron]

Obviously that would be amazing, and we'd love to help. One thing we can easily do, is get on a zoom call with a few engineers and answer your questions directly. @wicked dock can coordinate as needed.

[candid basin]

re: using a DaemonSet we just switched buildkitd to use a DS as well, for some of the same reasons but also to use PVCs. A colleague and I spent about 4 hours stracing containerd before finding that its disk utilization routine that runs every 10 seconds takes about 30 seconds to complete when it encounters buildkitd runc-worker snapshot directories. Using PVCs gets around it since they're omitted from metrics collection

[candid basin]

Right on. Thank you. I will definitely be in touch. I have to run this all by my manager and teammates first

[rare heron]

You know where to find us 🙂

[spiral vapor]

I'm struggling to find an example gitlab CI config that can run in a single job on the Kubernetes executor. Is there something self contained, or does it require a buildkit deployment?

[brisk fractal]

@spiral vapor I can provide you an example next week (as soon as I'm back from holidays) But yes, it requires a buildkit deployment. I recommend you to use the official Dagger Helm chart as starting point

[spiral vapor]

Thanks, good to know!

[spiral vapor]

Would love to know which helm chart is considered the official one, looks like docs online only reference the old dagger

[rare heron]

This is official: https://docs.dagger.io/194031/kubernetes

[spiral vapor]

Run Dagger on Kubernetes | Dagger

[timid yacht]

Slim hope at a guess, but worth an ask: anyone tried Dagger with dind running in sysbox (https://github.com/nestybox/sysbox )?

[signal sleet]

Yes but it doesn't work. We had it working initially in 0.8 but without dagger services enabled. This is not possible anymore with 0.9, so we switched to a dedicated box for our dagger engine...

[timid yacht]

Oh interesting. What wasn't working exactly? Do you recall the errors you encountered?

[rare heron]

Also interested 👀

[signal sleet]

So I just retried using sysbox, and got the following error:

time="2023-12-12T09:19:13Z" level=info msg="detected mtu 1500 via interface eth0"
dnsmasq[89]: started, version 2.89 cachesize 150
dnsmasq[89]: compile time options: IPv6 GNU-getopt no-DBus no-UBus no-i18n no-IDN DHCP DHCPv6 no-Lua TFTP no-conntrack ipset no-nftset auth no-cryptohash no-DNSSEC loop-detect inotify dumpfile
dnsmasq[89]: warning: interface dagger0 does not currently exist
dnsmasq[89]: using only locally-known addresses for dagger.local
time="2023-12-12T09:19:13Z" level=debug msg="setting up engine tracing"
dnsmasq[89]: reading /etc/dnsmasq-resolv.conf
dnsmasq[89]: using nameserver 172.20.0.10#53
dnsmasq[89]: using only locally-known addresses for dagger.local
time="2023-12-12T09:19:13Z" level=error msg="failed to create tracer provider" error="cannot merge resource due to conflicting Schema URL"
dnsmasq[89]: read /var/run/containers/cni/dnsname/dagger/addnhosts - 0 names
time="2023-12-12T09:19:13Z" level=debug msg="creating engine GRPC server"
time="2023-12-12T09:19:13Z" level=debug msg="creating engine lockfile"
time="2023-12-12T09:19:13Z" level=debug msg="creating engine controller"
time="2023-12-12T09:19:13Z" level=error msg="failed to create tracer exporter" error="cannot merge resource due to conflicting Schema URL"
time="2023-12-12T09:19:13Z" level=info msg="auto snapshotter: using overlayfs"
time="2023-12-12T09:19:13Z" level=debug msg="could not read \"/var/lib/dagger/net/cni\" for cleanup: open /var/lib/dagger/net/cni: no such file or directory"
time="2023-12-12T09:19:13Z" level=debug msg="creating new network namespace yk686y8k59878nrcavwq2cto2"
buildkitd: permission denied

so it fails at the CNI level, when trying to create a new network namespace

[rare heron]

@signal sleet would you mind opening an issue? 🙏

[signal sleet]

ok done: https://github.com/dagger/dagger/issues/6255

[rare heron]

Thanks, I responded there

[timid yacht]

This is useful to see, thanks for sharing. I haven't tried setting up dind on sysbox yet, it's on my list to do this month

[warm turtle]

interesting... since sysbox supposedly enables docker-in-docker which I assume uses network namespaces also. Definitely something to investigate since it sounds it should be possible

[signal sleet]

sysbox has known limitations: https://github.com/nestybox/sysbox/blob/master/docs/user-guide/limitations.md

in our case we got a "permission denied", so I was wondering if it was due to the binfmt-misc limitation, but I didn't spend more time investigating it. I'll add that link to the issue

[brisk fractal]

We suddenly started getting errors for our pipelines that have dind setup this morning from Gitlab's shared runners.

40: [4.25s] ip: can't find device 'nf_tables'
22740: [4.25s] modprobe: can't change directory to '/lib/modules': No such file or directory
22840: [4.26s] [WARN  tini (11)] Tini is not running as PID 1 and isn't registered as a child subreaper.
...
40: [21.5s] failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to create NAT chain DOCKER: iptables failed: iptables -t nat -N DOCKER: iptables v1.8.10 (nf_tables): Could not fetch rule set generation id: Invalid argument

Anyone else having issues?

All our pipelines that don't have dind still work fine.

And we haven't made any changes to the pipelines during past days.

[rare heron]

Was anything updated? Dagger, Docker, Kubernetes, Gitlab runner, Linux kernel?

[brisk fractal]

afaik no changes. BUT we use the shared runners so we can't control those really. I'll check if they are now ran in different runner than the past jobs that have succeeded.

Failing today in same runner as yesterdays passed job 🤔

Also Docker version is same.

20.10.16

so first line in logs that is different between fail and successfull is actually:

[20.4s] time="2023-12-15T08:32:02.407817074Z" level=info msg="unable to detect if iptables supports xlock: 'iptables --wait -L -n': `iptables v1.8.10 (nf_tables): Could not fetch rule set generation id: Invalid argument`" error="exit status 4"

That comes after Loading containers: start

[brisk fractal]

ah, we've been using the dind image without any specific version and seems it's been updated last night. So I think this is the problem.

[brisk fractal]

so docker:24.0.6-dind doesn't have this problem but the docker:24.0.7-dind has

(not gitlab related issue, sorry for spamming here but first suspected that it's about runners)

[slim warren]

Hi all,
I've hit a major issue.
<context>
I was SO happy to discover dagger a few day ago.
Started to figuring stuff out locally and I were still happy.
So I setup a gitlab-ci and pushed - then I've been friendly reminded by the error message that we do not allow dind at our companies shared runners for security reasons.
No big deal, just to go with kaniko - I THOUGHT.
Yes, no, it IS a big deal.
I found no way how to adapt dagger to use kaniko.
Private runners are not an option here.
</context>
Please, I really would like to go with Dagger, so please tell me there is a solution out there to go with kaniko instead of dind.

[signal sleet]

no there's no way to run dagger with kaniko - because the dagger-engine is a docker container itself.
if you really can't go with a custom gitlab runner, there's another alternative: setup a VM somewhere with docker, and expose the docker socket (with TLS...) - and configure dagger to connect to the remote docker socket to run its dagger-engine container.

[slim warren]

sorry but this is even worse than private runners.
this really hurts and we are probably not alone with this "dind sec-restrictions" thing
really hurts to leave dagger that soon, its a great approach with even greater capabilities 😭

[signal sleet]

I can confirm that you're not alone, we have the same issue - shared runners that won't work out of the box with dagger.
In our case, the issue is that we're using sysbox and it won't work with dagger.
But we've been able to still use dagger, 1st by using a remote docker daemon, and then by using a custom gitlab runner.
you could try to talk to the team in charge of the shared runners, and showcase what you can do with dagger - but I guess that you already know that 😉

[slim warren]

Unfortunately there is no much to talk with them - might be a sec-risk = No.
Even opening ports to pull (outgoing requests!) data from a on-prem db was not possible.

[normal ruin]

why did you opt to use a custom runner? I'm considering using a dedicated server(s) to run the dagger engine

[wicked dock]

Yeah unfortunately there is no great solution right now. 😦

There are two main constraints that require elevated permissions: https://github.com/dagger/dagger/blob/main/core/docs/d7yxc-operator_manual.md#can-i-run-the-dagger-engine-as-a-rootless-container

[timid yacht]

That docs talks about recent kernel versions... Is this in Dagger's backlog at all? dind isn't that popular with security teams (as Vincent/Cobalt/myself have all experienced), but since runner version 16.5 (I think?) dind-rootless is available and super easy to configure and use. I can see a couple of benefits to Dagger here: Gitlab is one of the top git/cicd providers, getting Dagger working on what I can only presume will be come one of the more popular ways of running Docker (due to simplicity) grows the potential userbase, and docker+machine/docker-autoscaler executor users are going to be looking at Dagger Cloud as an option?

[rare heron]

I believe rootless configurations don't work with Dagger out of the box, but I could be wrong

[timid yacht]

Certainly seems to be the case. My question is, is this on your roadmap/backlog? Feels like it's always going to be a blocker for Gitlab users running any of the docker executors and dind-rootless, and if I had to guess dind-rootless is going to replace dind for most users

[rare heron]

If it's easy to ship rootless support, of course we will ship it. If it's difficult (for example to support Dagger networking APIs, which I understand are one source of complication), then priority will depend on whether there are viable alternatives.

The obvious alternative is to run Dagger in its own privileged container, and connect to it from the Gitlab runner. This is the recommended pattern in production today. This could be a Daemonset on your Kubernetes cluster, or a separate machine or cluster - it all depends on your operational and security requirements.

Some teams are perfectly fine having Dagger as a privileged pod on the same cluster as their CI runners. Others aren't, and so would require a separate set of machines just for Dagger.

In any case, supporting remote engines, and beyond that multi-machine clusters, is a higher priority than rootless (which IMO will always be a bandaid - rootless will never work seamlessly for all workloads)

[rare heron]

In any case: yes, properly supporting running Dagger in a production Gitlab stack is important to us

[timid yacht]

I haven't had enough time to sink into this but I do intend to attempt to get the Dagger engine running as a Gitlab CICD service (and adding it to allowed_privileged_services). If I get time in the next week or two I'll report how I get on, I think it could be useful for others

[rare heron]

that would be super useful, thanks!

[timid yacht]

Started this last night, picked it up again briefly this morning. Number of early challenges:

Healthcheck (described here: https://docs.gitlab.com/ee/ci/services/#how-the-health-check-of-services-works) doesn't play nice with the Dagger engine. Don't think this is an issue beyond the warning based on how the healthcheck works.

*** WARNING: Service registry.dagger.io__engine-1 probably didn't start properly.
Health check error:
service "registry.dagger.io__engine-1-wait-for-service" health check: exit code 1
Health check container logs:
2024-01-31T09:49:11.147224093Z FATAL: No HOST or PORT found 

Storage driver for services. I thought this could be set with a global variable in the .gitlab-ci.yml file such that it applies to services as well as jobs, but a global variable didn't resolve that, guessing at this stage that it's only affecting docker:* services (no such errors in the docker service container logs):

2024-01-31T10:03:35.572546772Z time="2024-01-31T10:03:35Z" level=debug msg="auto snapshotter: overlayfs is not available for /var/lib/dagger, trying fuse-overlayfs: failed to mount overlay: invalid argument"
2024-01-31T10:03:35.572630966Z time="2024-01-31T10:03:35Z" level=debug msg="auto snapshotter: fuse-overlayfs is not available for /var/lib/dagger, falling back to native: fuse-overlayfs not installed: exec: \"fuse-overlayfs\": executable file not found in $PATH"
2024-01-31T10:03:35.572642657Z time="2024-01-31T10:03:35Z" level=info msg="auto snapshotter: using native"

Then the real problem - starting the engine times out:

1: connect
1: > in init
1: starting engine 
1: starting engine [600.0s]
1: connect ERROR: new client: context deadline exceeded
Error: new client: context deadline exceeded

The engine should be started on job start and clearly isn't, and that's unrelated to any of the job - it should be running the Dagger engine default entrypoint/cmd

[vocal venture]

👀 why is overlayfs not available?

[timid yacht]

At first I thought it was and the issue was with Gitlab not applying DOCKER_DRIVER=overlay2 but now I'm not so sure. Going to retest that one now with environment = ["DOCKER_DRIVER=overlay2"] in config and checking the module is present

Module was already available in system, already present in /etc/modules, and already loaded. Added environment to config.toml and service logs now show:

Health check container logs:
2024-01-31T10:30:28.727965285Z FATAL: No HOST or PORT found                      
Service container logs:
*********

Which suggests the service starts up fine but the healthcheck fails (expected)... Looks like engine is going to timeout again however.

[timid yacht]

Debugging from service logs, storage driver still isn't right:

[service:registry.dagger.io__engine-registry.dagger.io-engine-dagger] 2024-01-31T10:42:45.165871878Z time="2024-01-31T10:42:45Z" level=debug msg="auto snapshotter: overlayfs is not available for /var/lib/dagger, trying fuse-overlayfs: failed to mount overlay: invalid argument"
[service:registry.dagger.io__engine-registry.dagger.io-engine-dagger] 2024-01-31T10:42:45.165903468Z time="2024-01-31T10:42:45Z" level=debug msg="auto snapshotter: fuse-overlayfs is not available for /var/lib/dagger, falling back to native: fuse-overlayfs not installed: exec: \"fuse-overlayfs\": executable file not found in $PATH"

Docs (https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#use-the-overlayfs-driver) specify "when using dind", not sure how much that matters.

For the rest of the dagger engine logs I'd almost certainly need one of you to review, I have zero idea what should / should not be present. There's both service start-up log entries, and engine connection log entries

Starting engine log entries are all of the format:

[service:registry.dagger.io__engine-registry.dagger.io-engine-dagger] 2024-01-31T10:43:45.342951418Z time="2024-01-31T10:43:45Z" level=debug msg="engine metrics" cpu-count=2 cpu-idle=198596 cpu-iowait=9386 cpu-irq=0 cpu-nice=0 cpu-softirq=74 cpu-steal=2280 cpu-system=2269 cpu-total=217025 cpu-user=4420 dagger-server-count=0 disk-available-/=157956284416 disk-available-/var/lib/dagger=157956284416 disk-free-/=157973061632 disk-free-/var/lib/dagger=157973061632 disk-size-/=166304071680 disk-size-/var/lib/dagger=166304071680 goroutine-count=14 loadavg-1=0.43 loadavg-15=0.13 loadavg-5=0.22 mem-active=312668160 mem-available=7564029952 mem-buffers=51245056 mem-cached=770232320 mem-committed=1530286080 mem-free=6966820864 mem-inactive=727334912 mem-mapped=359972864 mem-page-tables=4857856 mem-shmem=1236992 mem-slab=94343168 mem-swap-cached=0 mem-swap-free=0 mem-swap-total=0 mem-total=8212094976 mem-vmalloc-used=16326656 proc-self-mem-anonymous=8294400 proc-self-mem-private-clean=29396992 proc-self-mem-private-dirty=8294400 proc-self-mem-pss=37691392 proc-self-mem-referenced=37695488 proc-self-mem-rss=37695488 proc-self-mem-shared-clean=4096 proc-self-mem-shared-dirty=0 proc-self-mem-swap=0 proc-self-mem-swap-pss=0 uptime=18m5s

Not sure how helpful that is

[warm turtle]

I think I know why this is the case @timid yacht. I have a hunch that this happens because you're running Dagger in a container and you're not setting up /var/lib/dagger to be a volume. Since you can't do overlay-in-overlay, the engine falls back to the fuse storage driver

[timid yacht]

So this can be tested by providing /var/lib/dagger as a volume? I've sort of moved away from trying to get Dagger running as a gitlab service in favour of just running it on a separate host, but happy to go back and test this

[warm turtle]

Yes, I'd try that out. Thing is that gitlab doesn't allow you to set volumes for services. There's an issue open about this in their repo

[timid yacht]

That doesn't surprise me, their services offering is skin-deep in other places as well

Can provide a shallow pool of use cases and not a lot else

[warm turtle]

I think there's something you can do to make it work though

If you create your own engine image from a Dockerfile and add the VOLUME instruction to it, that should work

[timid yacht]

Is there an example of that in the docs/repo somewhere?

[warm turtle]

We should do the same for our official image. I'll open an issue for that later today

[warm turtle]

No, there isn't but it's very straightforward.

Create a Dockerfile with the following contents:

FROM registry.dagger.io/engine:v0.9.10
VOLUME /var/lib/dagger

Build the image from that Dockerfile and try to use that instead

Curious to know if that will fix it if you can give it a try

[timid yacht]

Will do. It's nearly 11pm here so will be tomorrow now

[timid yacht]

Have made a start on this @warm turtle. Rather than keep spinning up instances manually on either my or work machines when I want to test/demo something I've gone ahead and wrote the terraform for it. Unsure how far I'll get with it today, probably just get both instances userdata done and leave the Dagger/Gitlab testing until tomorrow/Monday

[warm turtle]

Awesome! Let me know how it goes!

[faint wharf]

Hello, I'm trying to get my dagger pipeline running in Gitlab, but I haven't been able to get things working.
I've copied the example https://docs.dagger.io/145912/ci#gitlab-ci however it doesn't seem to be connect to docker.

$ dagger run node dagger/test.mjs
new client: failed to run container: docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?.
See 'docker run --help'.
: exit status 125

These are using the normal shared runners.
Are the examples out of date? or is there something else that I'm missing?

Eventually I will want to use my own runner, but I wanted to start with the "simplier" case first

[warm turtle]

hey there! this should work just fine, let me do a quick check to make sure it's working on the latest dagger version. I have a public gitlab example repo to test this out

[warm turtle]

@faint wharf just bumped my example to use the latest version of Dagger and seems to be running fine. Repo is here: https://gitlab.com/marcosnils/dagger-ci-test. Last CI job is here: https://gitlab.com/marcosnils/dagger-ci-test/-/jobs/6197555651 and pipeline definition here: https://gitlab.com/marcosnils/dagger-ci-test/-/blob/main/.gitlab-ci.yml

[faint wharf]

Hmm. I'll play around with it some more later tonight. Thanks for the repo

[faint wharf]

The interesting part is that dagger can't seem to connect to the docker daemon, however I am able to start up docker containers outside of dagger

[warm turtle]

Yes I saw that, but it works ok with my pipeline 🤷

[faint wharf]

It looks ike its not using the DOCKER_HOST .. I can kick off a normal docker command which seems to use the DOCKER_HOST=tcp://docker:2376 however dagger is trying to connect to unix:///var/run/docker.sock

[warm turtle]

Weird.. can you share your complete .gitlab.ci file? Are you running this in the public runners?

[faint wharf]

Right now this is a in a private repo. I am using a shared runner though. The end goal is to move to our own runner, but I figured it was easier starting with a shared runner

Use Dagger SDKs in CI | Dagger

[warm turtle]

Strange.. have you checked if there's any significant difference between my gitlab.ci and yours?

[timid yacht]

Ok, that does seem to have resolved the storage driver issue - no such warning now. Unsure if the engine is going to start up however, seems to be pretty stuck throwing out logs of this format: #gitlab message

Can't see any actual errors in the service logs, wondering if the machine is just too low-spec. Going to retry with a larger instance

[timid yacht]

1: connect
1: > in init
1: starting engine 
[service:dagger] 2024-02-19T14:30:52.035645007Z time="2024-02-19T14:30:52Z" level=debug msg="engine metrics" cpu-count=4 cpu-idle=84679 cpu-iowait=3310 cpu-irq=0 cpu-nice=379 cpu-softirq=338 cpu-steal=1039 cpu-system=2900 cpu-total=98958 cpu-user=6313 dagger-server-count=0 disk-available-/=4468609024 disk-available-/var/lib/dagger=4468609024 disk-free-/=4485386240 disk-free-/var/lib/dagger=4485386240 disk-size-/=8132173824 disk-size-/var/lib/dagger=8132173824 goroutine-count=16 loadavg-1=1.39 loadavg-15=1.54 loadavg-5=3.08 mem-active=624304128 mem-available=15826071552 mem-buffers=37642240 mem-cached=2491957248 mem-committed=1673568256 mem-free=13516935168 mem-inactive=2122440704 mem-mapped=373649408 mem-page-tables=4792320 mem-shmem=1216512 mem-slab=180703232 mem-swap-cached=0 mem-swap-free=0 mem-swap-total=0 mem-total=16566628352 mem-vmalloc-used=21975040 proc-self-mem-anonymous=8921088 proc-self-mem-private-clean=31985664 proc-self-mem-private-dirty=8921088 proc-self-mem-pss=40906752 proc-self-mem-referenced=40910848 proc-self-mem-rss=40910848 proc-self-mem-shared-clean=4096 proc-self-mem-shared-dirty=0 proc-self-mem-swap=0 proc-self-mem-swap-pss=0 uptime=4m8s

Still seems stuck on that, even on a much larger machine.

I have the service aliased to dagger and _EXPERIMENTAL_DAGGER_RUNNER_HOST=docker-container://dagger set, so unsure I can progress further than this

I'm up for further testing here, now have this terraformed so it's much quicker. Would be good to get this working if it's possible, even if I think I'll end up on a separate host

[warm turtle]

@timid yacht can you jump to a quick #911305510882513037 to check this out?

since you're around I'd assume it'll be faster

[timid yacht]

I should be able to in a few minutes if you're still around

[warm turtle]

sure, ping me when you're free 🙌

[timid yacht]

Ready when you are @warm turtle

[warm turtle]

we managed to make it work. Thing is that when you use gitlab services (https://docs.gitlab.com/ee/ci/services/#define-services-in-the-gitlab-ciyml-file) they actually run in the host docker engine and not the one inside the pipeline. So what we had to do is basically add command: ["--addr", "0.0.0.0:12345"] to the service definition and then set the _EXPERIMENTAL_DAGGER_RUNNER_HOST=tcp://$alias:12345 so the pipeline can connect to it

[vocal venture]

what gitlab has two docker engines running?

[warm turtle]

you generally have one engine in the host VM where the pipeline and additional services run, and then optionally, the pipeline itself generally launches a dind engine for the pipeline specific containers

[timid yacht]

Hitting what looks like a Dagger error after updating a module to 0.9.10: Error: failed to load module config: failed to parse git config file: json: cannot unmarshal object into Go struct field Config.dependencies of type string

Not Gitlab-related however

[warm turtle]

have you validated that your SDK and engine are running in the same version?

[timid yacht]

I think they are. Module has "engineVersion": "v0.9.10", Dockerfile has FROM registry.dagger.io/engine:v0.9.10

Runs locally on my 0.9.10 engine just fine which has puzzled me a bit, still trying to determine the problem

(Although I now need to add stdout to get the actual output?)

[rare heron]

unmarshal error

[flat umbra]

Hi there.
I’m new to dagger and try my first steps with gitlab-ci.
I wonder if I can avoid docker in docker approach as exposed here: https://docs.dagger.io/integrations/734201/gitlab
I read this quick exchange as a first entrypoint: https://news.ycombinator.com/item?id=38397548
So, did anybody try running dagger in Gitlab-ci without dind ?

marcosnils

Hi there, Dagger contributor here. We're solving this exact same problem by allowing you to encode your CI/CD pipelines in your favorite programming language and run them the same way locally and/or any CI provider (Gitlab, Github Actions, Jenkins, etc).We're very active in our Discord server if you have any questions! https://discord.gg/invite/...

Hacker News · Nov 23, 2023, 9:06 PM

[timid yacht]

@flat umbra I've spent quite a bit of time on this.

Short answers:

1. Use dind, dind-rootless even with the Dagger engine allowed to run privileged fails on provisioning, it needs dind to be privileged to provision.
2. Run the service yourself (needs a slightly customised Dockerfile). Has the advantage of not needing dind/dind-rootless, you lose a load of Dagger caching, but it does work and doesn't require that much setup
3. Run the service on another instance, point at it in your .gitlab-ci.yml (this is what I'm currently doing)

To expand on 2, if you want to get something working quickly, the Dockerfile you'll want is:

FROM registry.dagger.io/engine:v0.10.2
VOLUME /var/lib/dagger
CMD ["--addr", "tcp://0.0.0.0:12345", "--addr", "unix:///run/buildkit/buildkitd.sock"]

Push that to an image registry your runners can access, then in your .gitlab-ci.yml add:

variables:
  _EXPERIMENTAL_DAGGER_RUNNER_HOST: tcp://dagger:12345

services:
    - name: dagger-engine-image-above
      alias: dagger

(Credit to Marcosnils who helped me a ton with this above 👆!)

[flat umbra]

Thx a lot @timid yacht , that’s good to know, I’ll have a try on your solution 2.
I’m not used to run services in gitlab-ci, I’ll discover.
From what I’ve read, dagger cache is a must have, a main dagger feature, isn’t it ?

@timid yacht : I’d be very interested in knowing more in your third solution.

[timid yacht]

Dagger caching speeds up your pipelines considerably, and you will lose that benefit if you run the Dagger engine as a service (as described above)

The third option is just the same thing (run engine yourself) on another instance, then have your gitlab runners communicate with that instance

docker run \
  -v /var/lib/dagger \
  -p ${dagger_engine_port}:${dagger_engine_port} \
  --rm \
  --privileged \
  --name dagger-engine \
  registry.dagger.io/engine:v0.10.0 \
  --addr tcp://0.0.0.0:${dagger_engine_port} \
  --addr unix:///run/buildkit/buildkitd.sock

This is how I'm starting the Dagger engine on one instance, note both -p ${dagger_engine_port}:${dagger_engine_port} and --addr tcp://0.0.0.0:${dagger_engine_port} so the engine listens on that port, and that port is mapped to the host from the container

[flat umbra]

OK, so I imagine to deploy a dagger engine instance in a k8s cluster and then use it efficiently from anywhere (CI/CD or from my own CLIs): https://docs.dagger.io/integrations/104820/kubernetes/

Then, that’s not clear to me how to call a dagger engine from a gitlab-ci job.
Any idea ?

[vocal venture]

so i'd use the instructions in https://docs.dagger.io/integrations/734201/gitlab/ as a base for the gitlab side
essentially, what you'd want is to set the env variable _EXPERIMENTAL_DAGGER_RUNNER_HOST to the right kube-pod value as specified in the guide you already linked to: https://docs.dagger.io/integrations/104820/kubernetes/
(you also wouldn't need the dind service)

then, all future dagger cli invocations will go to that kubernetes pod

[faint wharf]

Something that I just figured out that may help other 'Gitlabers'
When a MR is created / updated, Gitlab creates a couple refs.

refs/merge-requests/${mr}/head    - points at head of branch
refs/merge-requests/${mr}/merge   - the merge commit of the entire MR
refs/merge-requests/${mr}/train   - when its on the merge train

I needed to be able to see the changes in the MR, so I needed to change how I checked out the code.

const ref = `ref/merge-requests/${mr}/merge`
const code = dag.git(repoUrl, {keepGitDir: true}).branch(ref).tree()
const container = dag
  .container()
  .from('node:20')
  .withDirectory('/src', code)
  .withExec(['git', 'remote', 'set-url', 'origin', repoUrl])
  .withExec(['git', 'fetch', 'origin', ref, '--depth=2'])

After that I was able to successfully use

container.withExec(['git', 'diff-tree', '--no-commit-id', '--name-only', '-r', 'HEAD^1', 'r', 'HEAD'])

To get a list of all of the changed files in the MR

[vocal venture]

fyi, the equivalent on github is refs/pull/${pr}/head and /merge
you can see all the refs, using git ls-remote (i was plumbing git internals into dagger at one point 😆)

[faint wharf]

Good to know... We will be moving over to GH in the near future (and dagger is going to make that much easier)

[maiden cloud]

👋 Since now GitLab is supported to host our modules, I'm planning an implementation where I work that emulates what we have in our Daggerverse publicly available repositories, but in GitLab private projects (project = repositories).
What I'm thinking in short is to have like my org/namespace/subgroup/dagger verse hosting my dagger modules that'll be installed and reused (dagger install) from other git lab's projects. Is there any specific restriction in GitLab to set this up? What's the recommendation, if any?

[vocal venture]

I think you'll just need to setup the subgroups in gitlab, and create the daggerverse repository? Or you could also create a daggerverse sub-group, and then each project/repo could potentially be a module 🤔

but shouldn't need any special gitlab setup, it should just work ™️

[maiden cloud]

Gotcha, I think so too. I'm planning to have a Daggerverse mono repository per SDK, it'd start with Python and Go since they're mainly the programming languages used internally.

[vocal venture]

any reason to do per-language? there should be no reason to not allow mix-matching in a single daggerverse.

[maiden cloud]

I initially thought that managing dot files and language-specific configurations would be more complicated if we mix linters, DX-tooling, etc., all in a single repository. However, I've changed my mind.

[wicked dock]

It can be a bit painful to have multiple go modules to get all the IDE stuff to work, using go.work sometimes works for me but often times I find the best way to get things to work is to open the subdir on its own.

[rare heron]

FYI discussion of possible future improvements to Dagger's prod architecture on Gitlab, using a real-world example https://discord.com/channels/707636530424053791/1274047894239973437

[vocal swift]

🙌 is there anyone here experience using dagger engine on kubernetes side by side gitlab runners? I need to create a demo next week. Some tips and tricks could be useful

[granite girder]

See the thread mentioned above

[jagged knot]

Hello folks, I'm trying to use a private daggerverse in gitlab and reference modules via dagger call -m ssh://gitlab.com/.... however the daggerverse is in a private subgroup (the company has a quite complex gitlab structure). When dagger tries to retrieve the module it fails with:

 ! failed to fetch remote ssh://[e-mail]/<company>/<subgroup>: git error: exit status 128
stderr:
remote: 
remote: ========================================================================
remote: 
remote: ERROR: The project you were looking for could not be found or you don't have permission to view it.
remote: 
remote: ========================================================================
remote: 
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.

and this is a very common error we faced while accessing private go modules. Gitlab documents the workaround here: https://docs.gitlab.com/ee/user/project/use_project_as_go_package.html#authenticate-git-requests-to-private-subgroups
However I believe dagger is not using any .netrc file in the host (when available).

Does anyone already faced this issue and has some hint how to solve it?

[rustic condor]

I jsut got this working, but maybe its not reflecting the structure correct?

mageep - subgroup (private)
dagger - subgroup (private)
datter-private-hello (git repo with module at the root)
dagger call -m ssh://[e-mail]/mageep/dagger/dagger-private-hello.git

i tried without .git and i got your error

[maiden cloud]

@vocal venture , kinda related to this ?https://github.com/dagger/dagger/blob/3f3a4a8a4d67f134e5fd3671205901f97cd2e719/core/schema/modulesource_test.go#L178

The project you were looking for could not be found or you don't have permission to view it
It sounds like an GitLab Group/Subgroups/project permissions error rather than how Dagger is handling the Git reference and parsing the Git endpoint.

@rustic condor , interesting. It seems that the presence or absence of .git at the end of the Git endpoint is handled gracefully:

     // Trick:
     // as we removed the scheme above with `parseScheme``, and the SCP-like refs are
     // now without ":", all refs are in such format: "[git@]github.com/user/path...@version"
     // transport.NewEndpoint parses users only for SSH refs. As HTTP refs without scheme are valid SSH refs
     // we use the "ssh://" prefix to parse properly both explicit / SCP-like and HTTP refs
     // and delegate the logic to parse the host / path and user to the library
     endpoint, err := transport.NewEndpoint("ssh://" + schemelessRef)
     if err != nil {
         return parsedRefString{}, err
     }
 
     gitParsed := parsedRefString{
         modPath:     endpoint.Host + endpoint.Path,
         kind:        core.ModuleSourceKindGit,
         scheme:      scheme,
         sshusername: endpoint.User,
     }

But, better await @vocal venture thoughts.

[rustic condor]

also in your message, ! failed to fetch remote ssh://[e-mail]/<company>/<subgroup>: git error: exit status 128

Its not showing the full path

this is what mine looks like also, when i removed .git

[vocal venture]

the .git in the repo name is significant for gitlab i think

cc @hollow dirge

the little "trick" mentioned @maiden cloud is wrong, but for a slightly more subtle reason to do with custom hosted git servers most of the time (aiming to fix this at some point)

[hollow dirge]

Yes, gitlab purposely dies not return the proper root of repo with private repos / subgroups

Link: https://gitlab.com/gitlab-org/gitlab-foss/-/blob/master/lib/gitlab/middleware/go.rb#L229-237

[hollow dirge]

Hi, when playing with GitLab refs linked to a private group or repo, the .git is necessary yes. Updated docs: https://github.com/dagger/dagger/pull/8346/files#diff-154322bef3eeb2f0d6cd6574f57ca3a8852066750f5ec6d91462764ad38aec8aR34

[jagged knot]

thanks for helping out, I'm able now to use private daggerverse with gitlab

[manic mountain]

Quick Q. Was looking at the example gitlab config and noticed that it ran everything in an alpine container and used docker in docker as a service; is there any reason for this? When I do docker stuff I usually just run directly in the docker in docker container

[rustic condor]

I think that works fine too. as long as the runner get install dagger

[rare heron]

Is this the example from the Dagger docs? Or another?

[manic mountain]

example from the docs; turns out you do need the service, as just using the dind container doesn't spin up the docker service so you can't run containers

however the example doesn't actually work as the alpine image doesn't have a docker client so it blows up when the dagger cli tries to run the engine

[rare heron]

Do you mind sharing the link so I make sure I'm looking at the right thing?

[manic mountain]

https://docs.dagger.io/integrations/gitlab/

.docker:
  image: docker:${DOCKER_VERSION}-dind
  services:
    - docker:${DOCKER_VERSION}-dind
  variables:
    DOCKER_HOST: tcp://docker:2376
    DOCKER_TLS_VERIFY: '1'
    DOCKER_TLS_CERTDIR: '/certs'
    DOCKER_CERT_PATH: '/certs/client'
    DOCKER_DRIVER: overlay2
    DOCKER_VERSION: '20.10.16'
    
.dagger:
  extends: [.docker]
  before_script:
    - apk add curl
    - curl -fsSL https://dl.dagger.io/dagger/install.sh | BIN_DIR=/usr/local/bin sh

That's what was needed to get it working

though I wonder if that could be simplified by using the dagger docker image and avoid the need to install dagger all together

[sour adder]

I'm trying to run basic coding checks, such as linting, unit tests and static analysis on a Gitlab CI pipeline.

Checks that pass consistently locally, are failing inconsistently on CI with this error:

Error: response from query: Post "http://dagger/query": unexpected EOF

Is anyone aware of what might be the issue? As well as suggestions on how I might debug errors like this in the future?

[sour adder]

My suspicion is I'm hitting some sort of resource limit. Less memory demanding jobs are less prone to this error. But it's inconsistent

[bronze creek]

I'm trying to run basic coding checks,

[vocal swift]

hey is there anyone here running dagger on gitlab with autoscaling feature? Currently all jobs scheduled to single node and stay in stuck state because of the lack of resources

[warm turtle]

hey is there anyone here running dagger

[iron condor]

I'm a bit confused about the behaviour I'm seeing when running Dagger on Gitlab CI.
I've following the instructions from the documentation
https://docs.dagger.io/integrations/gitlab#docker-executor
But for every run it seems to pull down the dagger engine Docker image and then pull my Docker image I'm running a test build in so it doesn't appear to be running any cache for Dagger. Has anyone else seen this and found a way to fix it?

[iron condor]

I've checked how it was running on our Github instance using the Github Action for Dagger, and that appears to run Dagger natively on the machine i.e. installs the dagger cli then the Dagger Docker image is run on the Github Actions Runner machine natively, instead of using Docker-in-Docker as with the recommended approach from the Gitlab example. Why is the Gitlab approach setup to use DinD?

[vocal swift]

GitLab CI | Dagger

[woeful wharf]

I'm testing out dagger in self-hosted gitlab with kubernetes executor. It seems the volume cache does not work. I can get it to work running locally against the engine in k8s (pointing to one of the pods in the daemonset with _EXPERIMENTAL_DAGGER_RUNNER_HOST). But in gitlab-ci it always executes the step. Is this expected?
I'm using the setup from here https://docs.dagger.io/integrations/kubernetes and from here https://docs.dagger.io/integrations/gitlab/#kubernetes-executor. The step I am using as test is .withMountedCache("/root/.npm", nodeCache) from the hello-dagger example (using typescript).

[vocal swift]

I'm testing out dagger in self-hosted

[rustic notch]

Hello Team
I am using dagger module for first time with engine version 0.13.0
module are now in gitlab repo,
i am able to run from local dagger cli with gitlab module url

[e-mail]:namespace/daggerverse.git/module@v0.0.13

but when i tried to run this from a gitlab runner which is connecting to remote dagger fleet server, it shows me this error

failed to resolve git tags: select: git command failed: exit status 128
Please make sure you have the correct access rights

Later, I logged in as root in that server to see if git has permission, though git clone of same module repository is fine and running dagger cmd with module is executing fine as well

Could you please guide what I am missing, i tried following https://docs.dagger.io/api/remote-modules#configuring-ssh-authentication
but not sure if I need to setup same because clonning module on dagger server directly is working fine there and manually running dagger cmd also working there

[maiden cloud]

Try to check this #gitlab message

[rustic notch]

I have followed that but still not working

According to the docs, https://docs.dagger.io/api/remote-modules/#authentication-methods
Dagger mounts the socket specified by your host's SSH_AUTH_SOCK environment variable to the Dagger Engine
so what exactly I need to mount to dagger engine and where?

[blissful shore]

hi guys im trying to run dagger engine locally on gitlab runner but i need on dagger publish for example

 dagger call publish --source=.
1   : connect
2   :   starting engine
3   :     create
4   :       exec docker run --name dagger-engine-v0.14.0 -d --restart always -v /var/lib/dagger --privileged registry.dagger.io/engine:v0.14.0 --debug

not to download or run dagger engine from dagger io but i got my own engine build in my harbor repo its possible to prevent dagger to automatically connect do default engine ? in which step of my gitlab pipeline i should use it ?
my pipeline example with sample values

.dagger:
  image: alpine:latest
  before_script:
    - export http_proxy=proxy.server:7777
    - export https_proxy=proxy.server:7777
    - export no_proxy=localhost
    - apk add curl docker
    - docker login $HARBOR_HOST -u $HARBOR_USERNAME -p $HARBOR_PASSWORD
    - curl -fsSL https://dl.dagger.io/dagger/install.sh | BIN_DIR=/tmp sh
    - export PATH=$PATH:/tmp

build:
  extends: [.dagger]
  variables:
    IMAGE_NAME: app
    IMAGE_TAG: 1.1.1
  script:
    - export http_proxy=proxy.server:7777
    - export https_proxy=proxy.server:7777
    - export no_proxy=localhost
    - export DO_NOT_TRACK=1
    - export OTEL_EXPORTER_OTLP_METRICS=""
    - dagger version
    - dagger call publish --source=.

[wicked dock]

Hey!

Yep, you should be able to use the _EXPERIMENTAL_DAGGER_RUNNER_HOST environment variable as described here: https://docs.dagger.io/configuration/custom-runner#connection-interface

[blissful shore]

thank u it helped now im using own engine and error changed 😄 moving to GO thread

[proud wagon]

Hey guys! Need help with configuring my .gitlab-ci.yml. I am currently getting following error

remote: HTTP Basic: Access denied. If a password was provided for Git authentication, the password was incorrect or you're required to use a token instead of a password. If a token was provided, it was either incorrect, expired, or improperly scoped. See https://gitlab.com/help/topics/git/troubleshooting_git.md#error-on-git-fetch-http-basic-access-denied
fatal: Authentication failed for 'https://gitlab.com/username/repo-name.git/'

here is my .gitlab-ci.yml file

.docker:
  image: docker:latest
  services:
    - docker:${DOCKER_VERSION}-dind
  variables:
    DOCKER_HOST: tcp://docker:2376
    DOCKER_TLS_VERIFY: '1'
    DOCKER_TLS_CERTDIR: '/certs'
    DOCKER_CERT_PATH: '/certs/client'
    DOCKER_DRIVER: overlay2
    DOCKER_VERSION: '27.2.0'
    # assumes the Dagger Cloud token is
    # in a masked/protected variable named DAGGER_CLOUD_TOKEN
    # set via the GitLab UI
    DAGGER_CLOUD_TOKEN: $DAGGER_CLOUD_TOKEN
.dagger:
  extends: [.docker]
  before_script:
    - apk add curl
    - curl -fsSL https://dl.dagger.io/dagger/install.sh | BIN_DIR=/usr/local/bin sh
hello:
  extends: [.dagger]
  script:
    - dagger -m https://oauth2:${GITLAB_PAT}@gitlab.com/username/repo-name.git call test --source=.

I am using oauth2:${GITLAB_PAT} as I have read thru few stackoverflow answers that PAT works when 2FA is enabled and we have to use oauth2 so that i runs correctly.
Can somepoint to the mistake I am making here or point me to some docs on how to get this working it would be great.

NOTE: I currently have not tried using ssh here.

Thanks!

[bronze creek]

Gitlab oauth2

[tacit shard]

Hi,

Just started looking at dagger with private modules in gitlab but I can't seem to get the dagger engine to pull modules with authentication. Here is my test setup running on private runners with docker-executor:

my-job:
  image: registry.gitlab.com/my-group/daggerverse:latest
  services:
    - name: docker:dind
  before_script:
    - git config --global credential.helper "store --file $GCM_PLAINTEXT_STORE_PATH"
    - echo "https://gitlab-ci-token:${CI_JOB_TOKEN}@gitlab.com" > $GCM_PLAINTEXT_STORE_PATH
  script:
    - cat $GCM_PLAINTEXT_STORE_PATH
    - git clone https://gitlab.com/my-group/daggerverse.git
    - find daggerverse/my-module
    - dagger -m https://gitlab.com/my-group/daggerverse.git/my-module call -h
  variables:
    GCM_PLAINTEXT_STORE_PATH: .git-credentials

The image contains docker cli (27), dagger cli (0.15.3) and git.
I am able to clone the repository but the dagger command returns an auth error.

[strong pike]

Hello, I read here and there how to run dagger jobs using the GitLab k8s executor. However I only saw configurations where the executor runs as root and mount the dagger socket (with a Daemonset making sure the dagger engine is available on every executor node).
I was wondering why so, rather than either binding to the tcp port or the kube-pod interface ? We try to avoid root executer on our CI platform, exposing only the dagger interface - if possible - sounds more secure.

[vocal swift]

Hello, I read here and [there](https://

[remote aurora]

Hello everybody.

I would like to ask for opinions. I just started exploring Dagger, and decided that after the first local demo it was time to integrate the Dagger Helm Chart's Daemonset in my Kubernetes cluster where I also run Gitlab (all in one Helm Chart... yes the one that is not advised for production).

So what I understood is basically that this DaemonSet places pods with the Dagger runtime on my cluster nodes, then for the rest it's all about configuring Gitlab in order to enable Dagger's CLI to reach the Dagger runtime through sockets (I don't know why in the examples an Alpine base image is used instead of an official Dagger or something).

I started to follow the kubernetes-executor documentation (https://docs.dagger.io/ci/integrations/gitlab/#kubernetes-executor), but I had a problem with defining the Runner, since installing Gitlab's all in one Helm Chart didn't download the Runner operator nor the CRDs for Runners (if I understood correctly it just created a deployment and that's it... I didn't dig much deeper than that into it), so I couldn't follow the guide strictly and found myself digging in the Helm values to insert the toml configuration and the GITLAB-RUNNER-TOKEN.

The only thing that seems a little odd is that I also had to create a certain amount of rbac RoleBindings for the first Hello Dagger pipeline to work because of the runner being in the "dagger" namespace and not "gitlab", and I guess that I will need to add more when I'll try out more elaborated modules...

Did I miss something stupid that allows for a much cleaner solution?
I am thinking about trying to disable the runners and install them standalone with the operator, maybe doing this would allow me to follow the documentation more strictly, avoiding potential issues.

[normal ruin]

Hello everybody.

[remote aurora]

CICD Jobs, one or many

Hi,
How do you structure your .gtlab-ci.yml file? I don't know what would be better:

• have one job that just calls a "do-everything" Dagger function
• have multiple jobs that call different steps of the Dagger Pipeline

Some context:
I am on a greenfield project that will not need crazy pipelines, just some basic:

• lint
• build
• test
• package
• push on registry
• maybe also a function to generate a dev environment, still not sure about this part

[rare heron]

Hello! This section of the docs has some advice for navigating that decision: https://docs.dagger.io/ci/adopting#integrating-with-ci

Specifically:

Daggerizing leads to simplifying. It's common to merge several large CI pipelines into a single one that just wraps dagger call. This usually leads to massive simplification of the CI configuration, as complex YAML/Groovy/shell spaghetti is replaced by clean code. Taken to the extreme, this process reduces the entire CI configuration to a single dagger call, with everything else happening inside Dagger. Although this sometimes happens, in practice most projects converge to a middle ground, where the CI configuration shrinks to just enough dagger call invocations to take advantage of proprietary CI features. Usually these features are job scaling, and job visualization. The more dependent you are on these proprietary CI features, the more granularity you will need to keep in your CI configuration.

[remote aurora]

Ok I will read this through, thanks!

My main concern right now is related to job visualization, as I am still figuring out how much I want to depend on Dagger Cloud to be able to read pipeline logs.

I also think that using Dagger Cloud more I will probably learn to appreciate its features better.

[rare heron]

yes looking into a detailed trace is definitely more pleasant than staring at a flat wall of text! It's worth familiarizing yourself with that feature. We use it for our own CI at Dagger and I basically never go to the CI interface anymore

In our case we do split in multiple jobs, but our constraint is job scale-out, not visualization

[remote aurora]

Noted.
Thanks for your precious feedback, the level of support you guys are giving me in these days is more than I would dare to hope for 😃

[remote aurora]

About using modules from remote repositories, I am trying to follow the documentation (https://docs.dagger.io/api/remote-modules/), but I cannot work my head around how to authenticate to my private gitlab.mydomain.com

I have created a project access token (stored in $PAT), but it does not work.
.gitlab-ci.yml:

.dagger:
  image: alpine:latest
  variables:
    DAGGER_CLOUD_TOKEN: $DAGGER_CLOUD_TOKEN
  before_script:
    - apk add curl git
    - curl -fsSL https://dl.dagger.io/dagger/install.sh | BIN_DIR=/tmp sh

build-dockefile:
  extends: [.dagger]
  script:
    - ls -latr
    - git ls-remote https://oauth2:$[e-mail]/shared/infrastructure/dagger-template-modules.git
    - dagger -m https://oauth2:$[e-mail]/sorcio-vibes/apps/familyexpensesplitter.git@v0.0.1 call \
      container-echo --source=. --args=./cmd/dagger \
      export --path=./build

The test git ls-remote works, but the dagger -m not.

$ dagger -m https://oauth2:$[e-mail]/sorcio-vibes/apps/familyexpensesplitter.git@v0.0.1 call \ container-echo --source=. --args=./cmd/dagger \ export --path=./build
...
11  : │ │ │ moduleSource(refString: "https://oauth2:[MASKED]@gitlab.mydomain.com/sorcio-vibes/apps/familyexpensesplitter.git@v0.0.1"): ModuleSource!
12  : │ │ │ │ GitRepository.tags: [String!]!
12  : │ │ │ │ GitRepository.tags ERROR [0.1s]
12  : │ │ │ │ ! git command failed: exit status 128
stdout: 
stderr: fatal: could not read Password for 'https://[e-mail]': No such device or address
11  : │ │ │ moduleSource ERROR [0.3s]
11  : │ │ │ ! failed to resolve git src: failed to resolve git tags: select: git command failed: exit status 128
stdout: 
...
stderr: fatal: could not read Password for 'https://[e-mail]': No such device or address

[remote aurora]

This is also confirmed in the Trace:
git command failed: exit status 128 stdout: stderr: fatal: could not read Password for 'https://[e-mail]': No such device or address

[wicked dock]

Hey!

Its going to use the git client to do authentication, so you you can't pass it in the URL like this

Did you try to do this using the glab auth method?

https://gitlab.com/gitlab-org/cli/-/blob/main/docs/source/auth/login.md

[remote aurora]

Yes I tried both:

test2:
  stage: test-dagger-module
  extends: [.dagger]
  before_script:
    - apk add curl git glab
    - curl -fsSL https://dl.dagger.io/dagger/install.sh | BIN_DIR=/tmp sh
    - glab auth login --hostname gitlab.mydomain.com --token $OWNER_USER_PASSWORD
  script:
    - dagger -m https://oauth2:$[e-mail]/shared/infrastructure/dagger-template-modules.git call container-echo --string-arg "hola test2"

I also tried (just putting the script command):

    - dagger -m https://gitlab.mydomain.com/shared/infrastructure/dagger-template-modules.git call container-echo --string-arg "hola test2"

And I tried both using a Project Access Token CICD variable or the account password for a bot user with owner permissions on the repository.

I keep getting errors like:

moduleSource(refString: "https://gitlab.mydomain.com/shared/infrastructure/dagger-template-modules.git"): ModuleSource!
11  : moduleSource ERROR [0.3s]
11  : ! failed to resolve git src to commit: failed to load cache key: error fetching default branch for repository https://gitlab.mydomain.com/shared/infrastructure/dagger-template-modules.git: git error: exit status 128
stderr:
remote: HTTP Basic: Access denied. If a password was provided for Git authentication, the password was incorrect or you're required to use a token instead of a password. If a token was provided, it was either incorrect, expired, or improperly scoped. See https://gitlab.mydomain.com/help/topics/git/troubleshooting_git.md#error-on-git-fetch-http-basic-access-denied

I guess that I could try with SSH, but I don't understand if I have to configure the SSH keys in the Gitlab CI Runner from where I am launching the $ dagger -m, or from the Dagger runtime (in a kubernetes cluster installed with he Helm Chart)

[strong pike]

Cross-posting here for awareness

[spice tree]

Hey guys!

I am running in a bit of a strange issue here .. and franky out of options.

My dagger pipeline does a simple thinng

        publish_syncer_container = await (
            dag.container()
            .build(context=syncer_folder)
            .with_registry_auth(syncer_image_registry, "token", token)
            .publish(f"{syncer_image_registry}-debug:{image_tag}")
        )

Token is a valid token with the needed permissions to read, write to the registry.
Now, when I run the code locally -> my image is published.
when I run the code via a gitlabCI using a Docker executor -> my image is published.
but when I am using the kubernetes executor -> the image is NOT published and this is the error :

Error: input: pythonFunctions.buildPublishSyncer failed to export: failed to push registry.gitlab.com/path/syncer-debug:syncer-v0.1.1: server message: insufficient_scope: authorization failed

The same token is used in the 3 scenarios.
Using a token with full permissions does not change a thing.

Any ideas on why running via the kubernetes executor could have an effect on authentication? 🤔

[rare heron]

👋 by any chance do you have a trace URL you could share?

[spice tree]

Salut @rare heron !
Yes sure. I have the runner logs regarding the publish part

[lusty jacinth]

What the Dagger Engine caching story like atm for GitLab? Does one exist?

[rigid rock]

I'm actually looking into this atm

[vocal swift]

I'm currently experimenting on this topic as well. The currently we're using long running self hosted gitlab runners(k8s) with dagger engine as a daemon set. It's not bad but handling gitlab concurrency is a bit issue.

[rare heron]

We're working on decoupling cache storage from compute in dagger ("project theseus"). Once that's done, the engine will be fully stateless and that will unblock a lot of fun scenarios for scaling it.

[rigid rock]

I ended up deploying the dagger engine helm chart, daemonset mode. So gitlab runner runs commands with the cli configured to use the Unix socket of the engine.

[rigid rock]

One thing I have noticed with gitlab ci, not sure if Github actions has the same issue, the output of running dagger call is not always captured, anyone else see this?

[lusty jacinth]

I wrote about deploying my Astro website to Cloudflare using GitLab CI with Dagger Shell

https://rawkode.academy/read/dagger-gitlab-cloudflare-deployment/

Now I'm hoping for a decent caching solution 😁

[blazing oriole]

Hoping for some help here. I create a dagger module with a bake-build function. The intent is to provide a reusable function/module for building images based on a docker-bake file. I'm trying to replace our existing Gitlab CI pipeline jobs that build from a docker-bake file and do this with a dagger based pipeline.

The problem I'm running into is that certain environment that are used in Gitlab CI are not available when the container build happens within dagger. And passing them each into the function is not practical because 1. We won't ever know in advance the variables a particular project might use in its Dockerfile for building and 2. passing each variable individually is cumbersome.

What's the right approach for this situation?

[rare heron]

Maybe you could take a .env file as argument?

There's no native support for .env files in the Dagger API (we discussed it, and might add it in the future) but it's easy to implement yourself in the module

[bronze creek]

yeah i was just going to link this issue: https://github.com/dagger/dagger/issues/8428
as a workaround, you could potentially pass in --env $(env) and load that in your build environment

[blazing oriole]

Looks like this gets a little more complicated since the env variables may contain special characters that are not allowed in the shell. bad variable name shows up far too often. I suppose I could grep out certain strings or characters based on a regex, but that just adds another layer of complexity that hopefully works right and doesn't encounter an unexpected character that breaks things later. It seems like I can't be the first one to run into this issue.

How do others handle this? If the modules are intended to be written in a way it is re-usable, or usable by several projects to repeat a single task (e.g. build from a docker-bake file), and since Dockerfiles may rely on any number of environment variables that can't all be known in advance, it seems like writing the entire .env file, sanitizing it, then sourcing it back in can't be the right way to do this?

[rare heron]

Taking a step back. You have a dagger function that performs a docker build. And you need to inject build arguments into that function - but you can't encode those build arguments as arguments to your function, because the build args are not known to you in advance.

Question: how does your build function work? Does it use Directory.dockerBuild() in the Dagger API? Or does it call the docker CLI from within the dagger function? And if the latter, how does it talk to the docker engine?

[blazing oriole]

The goal is to build based on a docker-bake file, so I don't think I can use dockerBuild() although I'm new to dagger so correct me if I'm wrong here. The bake file uses some environment variables that are present in the Gitlab environment, such as the URL to obtain python pip modules that will be installed. So I could inject that variable into the build function so it is available for this project's build process. But another project doesn't need that variable, and might need a different variable during the build process. I create the container with ctr := dag.Container() and any necessary "With..." and then run ctr.WithExec using docker buildx bake -f <bakefile> <targets>
I'm definitely open to suggestions if there's a better way to accomplish this. I've got a Gitlab runner in privileged mode and the Gitlab step is using a docker-in-docker service with TLS. I'm currently passing to the function:

1. The path to the bake file
2. The path to the source files
3. The targets to use from the bake file // Would be nice if I could default this to "default" if it isn't set
4. The path to the certs for TLS // Would be nice if I could default this since it will always be the same
5. The Service tcp://docker:2376 // Would be nice to default this since it will always be the same

[red mason]

I'm not even sure this has anything to do with Gitlab. I was looking into this issue and exec'd into the Kubernetes pod running the dagger call command, and I wasn't seeing any output there either. Maybe it has something to do with the dagger socket? Locally, I've always seen the output. It's only when we run dagger call within a kubernetes pod (setup with the dagger socket). The output is there sometimes, but it seems to be missing more than not. Maybe it's because there are so many containers involved? I.e. we use a kubernetes gitlab runner, wich fires up a dagger container, which starts a k3s cluster and then we run tests on that cluster. Too many turtles perhaps?

[red mason]

What I found facinating is that when I exec into the container, and run the dagger call multiple times, I end up getting output like every 4th or 5th attempt.

[sage spindle]

Hi, I am trying to run the hello-daggertemplate project on our self-hosted Gitlab CI (WIP upgrade infra to K8S).
https://docs.dagger.io/getting-started/ci-integrations/gitlab-ci/#docker-executor

On a Docker executor Runner, I encountered some failures with the provided gitlab-ci.yml (added the docker tags section to set the Docker based Runner).

About the certs file, not sure what's blocking. Coult it be that the Runner has no docker privilege ? 🤔

.docker:
  tags:
    - docker
  image: docker:latest
  services:
    - docker:${DOCKER_VERSION}-dind
  variables:
    DOCKER_HOST: tcp://docker:2376
    DOCKER_TLS_VERIFY: "1"
    DOCKER_TLS_CERTDIR: "/certs"
    DOCKER_CERT_PATH: "/certs/client"
    DOCKER_DRIVER: overlay2
    DOCKER_VERSION: "27.2.0"
    # assumes the Dagger Cloud token is
    # in a masked/protected variable named DAGGER_CLOUD_TOKEN
    # set via the GitLab UI
    DAGGER_CLOUD_TOKEN: $DAGGER_CLOUD_TOKEN

.dagger:
  extends: [.docker]
  variables:
    DAGGER_CLOUD_TOKEN: $DAGGER_CLOUD_TOKEN
  before_script:
    - apk add curl
    - curl -fsSL https://dl.dagger.io/dagger/install.sh | BIN_DIR=/usr/local/bin sh

test:
  extends: [.dagger]
  script:
    - dagger call test

test-e2e:
  extends: [.dagger]
  script:
    - dagger call e-2-e

build:
  extends: [.dagger]
  needs: ["test", "test-e2e"]
  script:
    - dagger call build

[sage spindle]

Saw this after searching through Discord: https://github.com/dagger/dagger/issues/4515

Will give a try on Gitlab.com

[sage spindle]

Yes, the dagger pipes work fine now on Gitlab.com 🙂

So this means :

• I'll need to ask our sefl-hosted gitlab instance Admin, to add a dedicated Dagger Runner (with docker-privileged setting for demo). Ops certainly prefer a rootless setting for security purposes, and an out of the box Dagger image (having engine + image build tools) would be ideal.

• I can further Daggerize our NodeJS projects

• I'll try to play with the Kube Executor on local/integration k3s cluster.

PS: Not sure why my DAGGER_CLOUD_TOKEN isn't valid, though I use the same token in CLI terminal and traces are available...

[rare heron]

@sage spindle we also have cloud-hosted engines in early access, if you're interested 😁 with automatic scaling and cache distribution

[sage spindle]

Sure, I'll look into it with our Ops Team 👀 🙂

[sage spindle]

I was wondering about :

1- the consistency between Dagger engine versions defined in project dagger.json VS gitlab-ci.yml (curl for installer) ?
I suppose we can update our project engine when Dagger CLI shows update message, to remain compatible.

2- the consistency between Docker engine versions defined in gitlab-ci.yml VS Gitlab-CI Runner settings ?

3-

the Dagger Engine is provisioned "just in time" using a Docker-in-Docker (dind) service.

the reason of having two docker images in .docker anchor : one for dagger engine hosting, and the other for image building (before pushing) ?
Why don't we use directly the same dagger container image (registry.dagger.io/engine:v0.20.0 we use on localhost) as job base image ?

Thanks

{
 "name": "hello-dagger",
 "engineVersion": "v0.20.0",
 "sdk": {
   "source": "typescript"
 },
 "source": ".dagger",
 "disableDefaultFunctionCaching": true
}

.docker:
  image: docker:latest
  services:
    - docker:${DOCKER_VERSION}-dind
  variables:
    DOCKER_HOST: tcp://docker:2376
    DOCKER_TLS_VERIFY: "1"
    DOCKER_TLS_CERTDIR: "/certs"
    DOCKER_CERT_PATH: "/certs/client"
    DOCKER_DRIVER: overlay2
    DOCKER_VERSION: "27.2.0"
.dagger:
  extends: [.docker]
  before_script:
    - apk add curl
    - curl -fsSL https://dl.dagger.io/dagger/install.sh | BIN_DIR=/usr/local/bin sh

 sudo gitlab-runner register -n \
    --name dagger \
    --url https://www.gitlab.com/ \
    --executor docker \
    --docker-privileged \
    --docker-volumes /cache \
    --docker-volumes /certs/client \
    --docker-image docker:20.10.16 \

[bronze creek]

hey 👋

the consistency between Dagger engine versions defined in project dagger.json VS gitlab-ci.yml (curl for installer) ?

As long as the engine is at least as new as the version in the dagger.json, you're good. So best is to keep the version in gitlab-ci.yml up to date with dagger releases, and update project code as needed for new features

Docker engine versions

I think as long as Docker is at least version 24 you're good

Why don't we use directly the same dagger container image

That gitlab reference in the docs is just an example, so yes there are lots of ways you could deploy dagger in gitlab. In that snippet, its setting up the runtime with the docker:latest base (but that really could be docker:cli), a dind service, and the dagger CLI. That way the CLI has the ability to provision its image at runtime to the dind service.

Its probably the simplest setup if you're using managed runners. But if you have control over your own CI infra you have a lot more options. For example, if you're running your gitlab CI runners in kubernetes, the runner pod could have a sidcar with the registry.dagger.io/engine:v0.20.0 and a socket mounted to the runner