Is it possible to make host devices available in a dagger pipeline? | Dagger | Page 1

shy wedge Oct 4, 2023, 8:56 AM

#

Hi!

My first experiments with dagger are very positive, the flexibility is superb, which makes it a good candidate for switching over multiple projects from gitlab-ci to dagger. (still executing dagger inside of gitlabci)

1 thing I am struggling with is the fact that is seems currently impossible to mount any /dev/ttyX devices inside of the pipeline containers.

I'm working for an embedded SW quality company, and basically most of our pipelines communicate with some programmers, serial ports and many other devices in linux..

I saw the following post on github: https://github.com/dagger/dagger/issues/3354 But it seems no real solution was present.. Of course, this was 1 year ago, so that's why I'm asking here..

Is there any other workaround that COULD possible be feasible?

Thanks in advance guys!

GitHub

✨ support mounting device into the build · Issue #3354 · dagger/dag...

What are you trying to do? I need to access /dev/ttyUSB0 to be able to flash some stm32 microcontroller. Once I built an app with tinygo, I need to use stm32flash on /dev/ttyUSB0 with it. Why is th...

shy wedge Oct 4, 2023, 2:50 PM

#

Is it possible to make host devices available in a dagger pipeline?

ocean magnet Oct 4, 2023, 5:48 PM

#

hey @shy wedge! I think there's a very temporary workaroud for this. Let me try something really quick

shy wedge Oct 5, 2023, 7:57 AM

#

@ocean magnet OK, very curious now, thx for trying already 😄

ocean magnet Oct 6, 2023, 12:03 PM

#

hey! apologies on the delay. My workaround didn't work as expected and I'm thinking about altearnatives. Will update my progress here 🙏

shy wedge Oct 6, 2023, 12:36 PM

#

Ok, nice thx! Just to be curious, what are you thinking about? Might be able to think along.. 🙂

ocean magnet Oct 6, 2023, 6:18 PM

#

there are some options you can pass to buildkit to configure the underlying OCI runner (https://github.com/moby/buildkit/blob/master/docs/buildkitd.toml.md?plain=1#L52). I was trying with the noProcessSandbox flag but then I realized that buildkit requires rootless if we're setting that flag

#

that's where I got initially stuck, but I still think there might be a way to configure the underlying OCI runtime to give access to specific devices

shy wedge Oct 9, 2023, 8:42 AM

#

@near olive I've heard from some other member you are a real expert on the Buildkit stuff, do you haver any clue or idea if what I need could be achieved in some way currently?

ocean magnet Oct 9, 2023, 2:39 PM

#

@shy wedge I briefly mentioned this to Erik during our offsite the last week and the answer is that given the current buildkit architecture, we don't have a straightforward way to support this. We acknowledge it's a requirement we should support soon.

Having said that, in order to unblock you so you can finish your testing, I can suggest a quick code edit you could do on the engine to support what you need

#

if you're ok with building your own engine image until we have the bandwidth to support this feature natively.

shy wedge Oct 11, 2023, 7:28 AM

#

@ocean magnet no problem at all with that! However you can give me the needed information, I'm happy to work with it 🙂

ocean magnet Oct 11, 2023, 1:33 PM

#

so, @shy wedge what you can do is edit this part of the code: https://github.com/dagger/dagger/blob/main/cmd/shim/main.go#L372-L377 and add another mount there with the device that you want to have in your dagger containers

GitHub

dagger/cmd/shim/main.go at main · dagger/dagger

A programmable CI/CD engine that runs your pipelines in containers - dagger/dagger

#

after you're done changing that, you can call ./hack/dev which will build a dev engine image version you can use locally and in your CI

#

here's an example I just did to mount a disk device:

        spec.Mounts = append(spec.Mounts, specs.Mount{
            Destination: "/dev/nvme0n1",
            Type:        "bind",
            Source:      "/dev/nvme0n1",
            Options:     []string{"rbind", "ro"},
        })

ocean magnet Oct 11, 2023, 1:55 PM

#

take into account that if that device doesn't exist all your Execs will fail, so you need to a condition to validate the device exists before adding the mount

shy wedge Oct 11, 2023, 2:39 PM

#

@ocean magnet OK perfect, thx! I will give it a try tomorrow and report back! 😄

shy wedge Oct 17, 2023, 12:21 PM

#

Hi @ocean magnet sorry for the delay, got sidetracked. I've managed to try it out! I added the following to the main.go spec.Mounts = append(spec.Mounts, specs.Mount{ Destination: shimPath, Type: "bind", Source: selfPath, Options: []string{"rbind", "ro"}, }) spec.Mounts = append(spec.Mounts, specs.Mount{ Destination: "/dev/ttyDEV1", Type: "bind", Source: "/dev/ttys004", Options: []string{"rbind", "ro"}, })
I made a socat virtual serial port link, so I locally on my machine (m1 Macbook pro) have the following ports: /dev/ttys002 and /dev/ttys004, which I am trying to link in using the snippet above. Unfortunately I get the following error:

#

54: exec apk add build-base go git ERROR: process "apk add build-base go git" did not complete successfully: exit code: 1 54: > in engine > dev 54: [0.06s] runc run failed: unable to start container process: error during container init: error mounting "/dev/ttys004" to rootfs at "/dev/ttys004": stat /dev/ttys004: no such file or directory 54: exec apk add build-base go git ERROR: process "apk add build-base go git" did not complete successfully: exit code: 1

ocean magnet Oct 17, 2023, 2:14 PM

#

hey @shy wedge! hmm I see you're using a mac. Not sure how this will work in docker for mac since there's another layer of virtualization which against the hypervisor to run docker

#

if you run docker run --rm --privileged alpine ls -la /dev do you see your devices there?

shy wedge Oct 17, 2023, 2:16 PM

#

Ah yes, true! Good point, don't see the devices there, either... I'll try to figure it out, thanks anyways already a ton!

ocean magnet Oct 17, 2023, 2:21 PM

#

shy wedge Ah yes, true! Good point, don't see the devices there, either... I'll try to fig...

sure, np!

shy wedge Oct 17, 2023, 2:31 PM

#

@ocean magnet OK, got it compiling now, after modifying the macos Docker settings, to allow mounting in /dev. But when executing a quick "Hello, World" I don't really see the device ``

📎 message.txt

#

Should I still mount the device in the python script itself then? I now have the following:

#

`import shlex
import sys
import anyio
import dagger

async def test():
async with dagger.Connection(dagger.Config(log_output=sys.stderr)) as client:
src = client.host().directory(".")
python = (client.container().from_(f"ubuntu").with_workdir("/src").with_exec(["ls", "-la", "/dev"]).with_exec(shlex.split("echo 'Hello' > /dev/ttyDEV1")))
await python.stderr()

anyio.run(test)`

ocean magnet Oct 17, 2023, 2:33 PM

#

you need to tell dagger to use your locally built engime image

#

are you setting the _EXPERIMENTAL_DAGGER_RUNNER_HOST variable?

shy wedge Oct 17, 2023, 2:34 PM

#

Ah yes, totally forgot.. In my previous experiments I used that, but for this one I forgot.. 😄 ┃ │ dagger.ExecError: process "ls -la /dev" did not complete successfully: exit code: 1 ┃ │ Stdout: ┃ │ ┃ │ Stderr: █◀──╯ [0.10s] ERROR exec ls -la /dev ┃ runc run failed: unable to start container process: error during container init: error mounting "/dev/ttys004" to rootfs at "/dev/ttyDEV1": stat /dev/ttys004: no such file or directory

#

I will have to try on a linux machine maybe..

#

Although in the Alpine image, when I mount in /dev explicitely, I can see the /dev/ttys004. But OK

ocean magnet Oct 17, 2023, 2:35 PM

#

maybe you need to restart your engine container

#

you should have an engine container running called dagger-engine.dev

#

can you do docker exec -ti dagger-engine.dev ls -la /dev? And check if the device is there

#

oh.. maybe it's not called dagger-engine.dev and has a different name. Can't recall exactly what the name is when you set a custom image

shy wedge Oct 17, 2023, 2:38 PM

#

I got one named, dagger-eninge.dev and one: dagger-engine-be7d2e4256ad2b56. Both don't have it in there by default

ocean magnet Oct 17, 2023, 2:38 PM

#

ok, try removing those containers

shy wedge Oct 17, 2023, 2:39 PM

#

ok

ocean magnet Oct 17, 2023, 2:39 PM

#

and running the pipeline again

#

maybe those containers were created before you added the /dev thing in your engine

#

the device needs to be present in that container for the engine to pick it up

shy wedge Oct 17, 2023, 2:41 PM

#

OK, will try it out. If it does not work, I'll try to find a linux machine to try it on 😄

ocean magnet Oct 17, 2023, 2:47 PM

#

if you see the device in the alpine image it should be possible to get them in the engine image

#

did you have to specifically add the -v flag to docker run to see them in the alpine container before?

shy wedge Oct 17, 2023, 2:52 PM

#

Yes!

#

Just wondering, but to make it a bit more efficient, could I not just mount the whole /dev in the main.go then..? Because also for re-use for any projects, just mounting a specific name is not going to give a very big advantage.. Then I would have to recompile the engine for all places it will run..

ocean magnet Oct 17, 2023, 3:02 PM

#

shy wedge Just wondering, but to make it a bit more efficient, could I not just mount the ...

you can but take into account that exposing the entire /dev will not very secure since you're exposing the underlying machine devices to the pipeline

#

you can also change the code so you can decide which device to mount when the engine runs

#

like MOUNT_DEVICE=/dev/foo dagger run .....

shy wedge Oct 17, 2023, 3:03 PM

#

Yes, I was indeed thinkin more about something like that..

ocean magnet Oct 17, 2023, 3:04 PM

#

shy wedge Yes!

did it work?

shy wedge Oct 17, 2023, 3:05 PM

#

No, that was just referencing that I had to explicitly add the -v to mount them in there

#

I added the complete /dev folder now, and I get a list of devices now, but it's far from complete.. I'm not sure if they even are the same devices from my machine tbh..

ocean magnet Oct 17, 2023, 3:08 PM

#

shy wedge No, that was just referencing that I had to explicitly add the -v to mount them ...

I see if that's the case you can start the engine yourself instead of letting dagger automatically start it

#

docker run -d --privileged --name dagger-engine.dev localhost/dagger-engine.dev:latest and you can add the device you want there with -v

#

you can then tell dagger to use that engine by setting _EXPERIMENTAL_DAGGER_RUNNER_HOST=docker-container://dagger-engine.dev

shy wedge Oct 17, 2023, 3:09 PM

#

Ah yes, good idea.. Lemme try that one

shy wedge Oct 18, 2023, 9:56 AM

#

@ocean magnet Could still be related to macos, but with the alpine container, this does work: `11:54:22 ➜ docker run --rm --privileged -v /dev:/dev --name dagger-engine-qtl2.dev localhost/dagger-engine.dev:latest
2023/10/18 09:54:24 splice: permission denied
panic: splice: permission denied

goroutine 1 [running]:
log.Panicf({0x1641026?, 0x163db19?}, {0x40000b4de8?, 0x1?, 0x0?})
/usr/local/go/src/log/log.go:439 +0x64
github.com/hanwen/go-fuse/v2/splice.init.1()
/go/pkg/mod/github.com/hanwen/go-fuse/v2@v2.2.0/splice/splice.go:57 +0x250
`

ocean magnet Oct 18, 2023, 2:35 PM

#

shy wedge <@336241811179962368> Could still be related to macos, but with the alpine conta...

👋 can you try mounting a single device instead of the whole /dev directory?

shy wedge Oct 19, 2023, 7:54 AM

#

@ocean magnet OK, I did it with just 1 port, then the engine starts up, and with ls -la, I can see it! If I echo something in it, though, I don't see it coming out on the host machine, but could be related to the "with_exec" parsing of `shlex.split("echo Hello > /dev/ttys010"). Will check that now..

ocean magnet Oct 19, 2023, 1:01 PM

#

shy wedge <@336241811179962368> OK, I did it with just 1 port, then the engine starts up, ...

does the echo work if you do it in the alpine container?

#

just trying to rule out it's not a docker for mac thing

vapid tiger Feb 27, 2024, 5:54 AM

#

Is the source patch above still the recommended way to achieve device mounts? And it doesn't require any .withX("/dev/foo", dag.host().x("/dev/foo"))?

I need to expose /dev/kvm on a Linux host.

ocean magnet Feb 27, 2024, 1:27 PM

#

vapid tiger Is the source patch above still the recommended way to achieve device mounts? A...

hi, yes unfortunately. Given that it's not a super common request, we haven't prioritized adding first class support for this in the engine given that it's not trivial and requires quite some work

#Is it possible to make host devices available in a dagger pipeline?