IPC/shared memory

[manic socket]

I have a service that utliizes IPC, tests run fine when I invoke the application with docker --ipc=host, is there a way to enable IPC for the dagger engine (specifically: creating entries in /dev/shm)? I'm running the engine as a daemonset via helm.

[edgy cliff]

👋 @manic socket to better understand the requirement, you're expecting your Dagger functions to be able to access the host's /dev/shm given that there are some services already running there which you need to communicate to?

is your pipeline fully daggerized? Are you running both your service and tests in Dagger?

[manic socket]

it's more like there's a test suite that contains tests that start a server and client that communicate with each other over IPC. no, the service is not instantiated via dagger (could be a good idea to do so)

[edgy cliff]

if there's a way you could start the service via Dagger, then you can use WithMountedCache between the service and its dependent tests so they all have access to the same volume for IPC

[edgy cliff]

another option which doesn't require that many changes if your IPC files are unix sockets is to define a *dagger.Socket type in your function arguments and then pass that.

This way, you can do something like:

dagger call myfunc --svc-socket unix://dev/shm/svc.sock or something along those lines

then in your Dagger function you can use the *dagger.Socket and mount it in your test clients wherever is needed

[manic socket]

that may cover more cases, but this code is calling shm_open directly, and creates multiples segments in parallel. I've tried enabling hostIPC in the pod spec for the engine and mounting /dev/shm as a ramdisk so far. I need to experiment a little more with this.

I'll try withMountedCache too

[edgy cliff]

that may cover more cases, but this code is calling shm_open directly, and creates multiples segments in parallel. I've tried enabling hostIPC in the pod spec for the engine and mounting /dev/shm as a ramdisk so far. I need to experiment a little more with this.

yeah.. this won't work because the even if you do that, the containers spawned by Dagger will still heave its own IPC share and you won't be able to use the host's.

So I think the only stopgap at this stage is running your service within dagger and then mounting a cache volume in /dev/shm so you can share that across multiple containers. The caveat here is that you can expect a considerable pentaly in performance given that by doing this /dev/shm won't be effectively backed by a tmpfs but an actual physical disk

[manic socket]

thanks for the feedback. the tests are sending >1G files across shared memory and this appears limited by the size of the /dev/shm mount. which is tmpfs from RAM. and I was able to make that volume larger for the dagger engine daemonset but the tests are actually running inside the gitlab-runner pod (build or service container) which has a 64M sized /dev/shm.

[edgy cliff]

@manic socket that's the shm mounted in the Dagger containers, correct?

[manic socket]

i'm not sure what you mean by dagger container.
there's the daemonset's dagger-engine pod
there's the gitlab-runner's pod, which has a "build" container where the dagger function is called from, and a "service" container which is docker:dind
the code running inside the dagger-engine is the dagger function code, in this container, the /dev/shm mount is modified to be larger

if i could reproduce this locally, this would be easier to troubleshoot, i'll work on modify my local dagger-engine container to expand beyond the 64M /dev/shm limit

[edgy cliff]

@manic socket everything that runs within your Dagger functions runs in containers. That's what I meant with "dagger container". Those containers don't have access to the Dagger engine /dev/shm mount path

It's what I was trying to explain here #1387542289685155951 message

[manic socket]

ok i'm doing this:
shmCache := dag.CacheVolume("shm-cache")
buildContainer = buildContainer.WithMountedCache("/dev/shm", shmCache, dagger.ContainerWithMountedCacheOpts{Sharing: dagger.Locked})
buildContainer = buildContainer.WithExec([]string{"/bin/bash", "-c", "df -h /dev/shm"})
but I'm getting:
df: /dev/shm: No such file or directory

dagger-engine logs

[edgy cliff]

hmm seems that cache volumes can't be mounted in /dev. Wondering if there might be a workadound here cc @paper solstice . Basically trying to make /dev/shm work across dagger containers

I'd assume that might be possible with an entrypoint hack script. Let me try something really quick

[paper solstice]

Yeah any mount attempt to /dev will likely get overwritten by the actual /dev mount, which currently happens later in the container setup process

[edgy cliff]

yeah.. this works 🙈

func (m *Shm) Test(ctx context.Context) (string, error) {
    return dag.Container().
        From("alpine:latest").
        WithNewFile("/entrypoint.sh", `#!/bin/sh
set -e
mount --bind /cache /dev/shm
exec "$@"`, dagger.ContainerWithNewFileOpts{Permissions: 0755}).
        WithEntrypoint([]string{"/entrypoint.sh"}).
        WithMountedCache("/cache", dag.CacheVolume("shm")).
        WithExec([]string{"sh", "-c", "df"}, dagger.ContainerWithExecOpts{UseEntrypoint: true, InsecureRootCapabilities: true}).
        Stdout(ctx)
}

[edgy cliff]

Erik cries in security 😛

[paper solstice]

at least the arg is named InsecureRootCapabilities 🙂

[paper solstice]

this probably is the best it'd get atm, but I can certainly imagine "configuring a shared /dev/shm" having better first class support in the API itself

[edgy cliff]

Yeah this came up with some other folks also. If there's enough demand, we can make it happen

@manic socket I'd try it this way and go from here

[manic socket]

// run tests.py via pytest, requires optional-dependencies defined for a "test" extra
func (m *Build) PythonTests(ctx context.Context,
    // build env containing all build tools and envvars
    buildEnv *dagger.Container,
    // git repo or local directory that contains the tests.py and pyproject.toml, should match project parameter
    packageRepo *dagger.Directory,
    // package registry with HTTP auth
    pypiUrl *dagger.Secret,
    project string,
    version string,
) *dagger.Container {
    uid, _ := buildEnv.User(ctx)
    buildContainer := buildEnv.WithUser("0").WithNewFile("/entrypoint.sh", `#!/bin/bash
set -e
(
echo in entrypoint
mount --bind /cache /dev/shm
exec "$@"
) 2>&1 | tee /tmp/entrypoint.log
`, dagger.ContainerWithNewFileOpts{Permissions: 0755}).WithEntrypoint([]string{"/entrypoint.sh"}).WithMountedCache("/cache", dag.CacheVolume("shm"))

    buildContainer = buildContainer.WithExec([]string{"/bin/bash", "-c", "chmod 1777 /cache ; df -h /dev/shm; ls -al /dev/shm"}, dagger.ContainerWithExecOpts{UseEntrypoint: true, InsecureRootCapabilities: true})
    buildContainer = buildContainer.WithUser(uid).WithSecretVariable("UV_EXTRA_INDEX_URL", pypiUrl)
    buildContainer = buildContainer.WithMountedDirectory("/build", packageRepo, dagger.ContainerWithMountedDirectoryOpts{Owner: uid}).WithWorkdir("/build")
    buildContainer = buildContainer.WithExec([]string{"/bin/bash", "-c", "(df -h /dev/shm; ls -al /dev/shm) 2>&1 | tee /tmp/stage2.log"})
    buildContainer.Terminal() // this doesnt work even when "dagger call -i" is ran?
    if version != "" && project != "" {
        buildContainer = buildContainer.WithExec([]string{"/bin/bash", "-c", "uv pip install " + project + "[test]==" + version})
    } else {
        buildContainer = buildContainer.WithExec([]string{"/bin/bash", "-c", "uv pip install --strict " + project + " .[test]"})
    }
    return buildContainer
}

having trouble getting the cache bind mount after switching to the non-root user

how did you get that nice formatting to work in discord?

anyway, first:
$ cat /tmp/entrypoint.log
in entrypoint
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/cl-docker--xfs 320G 232G 89G 73% /dev/shm
total 0
drwxrwxrwt 2 root root 6 Jul 2 15:54 .
drwxr-xr-x 6 root root 900 Jul 2 18:31 ..
so it worked
but:
$ cat /tmp/stage2.log
Filesystem Size Used Avail Use% Mounted on
shm 64M 0 64M 0% /dev/shm
total 0
drwxrwxrwt 2 root root 40 Jul 2 18:31 .
drwxr-xr-x 5 root root 340 Jul 2 18:31 ..

changing the debug commands to df -h /dev/shm; ls -ial /dev/shm /cache:

in entrypoint
Filesystem                  Size  Used Avail Use% Mounted on
/dev/mapper/cl-docker--xfs  320G  233G   88G  73% /dev/shm
/cache:
total 0
467973062 drwxrwxrwt 2 root root  6 Jul  2 15:54 .
576218181 drwxr-xr-x 1 root root 56 Jul  2 18:42 ..

/dev/shm:
total 0
467973062 drwxrwxrwt 2 root root   6 Jul  2 15:54 .
        1 drwxr-xr-x 6 root root 900 Jul  2 18:42 ..
dagger /build $ cat /tmp/stage2.log
Filesystem      Size  Used Avail Use% Mounted on
shm              64M     0   64M   0% /dev/shm
/cache:
total 0
467973062 drwxrwxrwt 2 root root  6 Jul  2 15:54 .
576218184 drwxr-xr-x 1 root root 43 Jul  2 18:42 ..

/dev/shm:
total 0
1 drwxrwxrwt 2 root root  40 Jul  2 18:42 .
1 drwxr-xr-x 5 root root 340 Jul  2 18:42 ..

without WIthUser("0"), it can't bind mount:

in entrypoint
mount: /dev/shm: must be superuser to use mount.

[edgy cliff]

adding "```go" to the snippet

[edgy cliff]

@manic socket how about installing su in the container so you can do su root mount... so that way the container can be executed with any user

[manic socket]

I mean, it's there and I can setup passwordless root escalation, but again at "stage2" the mount reverts back.
if I set insecureRootCapabilities: true, useEntrypoint: true on the "stage2" step, then I can see the mount. so maybe I need to set those for every EXEC?

appears to be the case! tests are passing and not SIGBUS'ing!

but I think I'd want the ability for someone to debug the tests via dagger call tests terminal . I think..

really? my base image is ubuntu and it looks like i have ubuntu tools in ther terminal

[edgy cliff]

nvm.. you're right. I was getting confused

[edgy cliff]

what I'm not sure about is if terminal actually runs the container entrypoint

can you check that?

you can also pass --insecure-root-capabilities to the terminal command

[manic socket]

ah let me try that

[manic socket]

no dice, this is what I've ended up with:

    uid, _ := buildEnv.User(ctx)
    buildContainer := buildEnv.WithUser("0").WithExec([]string{"/bin/bash", "-c", "passwd -d root"})
    buildContainer = buildContainer.WithUser(uid).WithNewFile("/entrypoint.sh", `#!/bin/bash
#set -e
#(
echo in entrypoint
su - root -c "mount --bind /cache /dev/shm"
exec "$@"
#) 2>&1 | tee -a /tmp/entrypoint.log
`, dagger.ContainerWithNewFileOpts{Permissions: 0755}).WithEntrypoint([]string{"/entrypoint.sh"}).WithMountedCache("/cache", dag.CacheVolume("shm"))

    buildContainer = buildContainer.WithExec([]string{"/bin/bash", "-c", "chmod 1777 /cache ; df -h /dev/shm; ls -al /dev/shm"}, dagger.ContainerWithExecOpts{UseEntrypoint: true, InsecureRootCapabilities: true})
    buildContainer = buildContainer.WithUser(uid).WithSecretVariable("UV_EXTRA_INDEX_URL", pypiUrl)
    buildContainer = buildContainer.WithMountedDirectory("/build", packageRepo, dagger.ContainerWithMountedDirectoryOpts{Owner: uid}).WithWorkdir("/build")
    if false {
        buildContainer = buildContainer.WithExec([]string{"/bin/bash", "-c", "(df -h /dev/shm; ls -al /dev/shm) 2>&1 | tee /tmp/stage2.log"}, dagger.ContainerWithExecOpts{UseEntrypoint: true, InsecureRootCapabilities: true}) )
    }
    if version != "" && project != "" {
        buildContainer = buildContainer.WithExec([]string{"/bin/bash", "-c", "uv pip install " + project + "[test]==" + version})
    } else {
        buildContainer = buildContainer.WithExec([]string{"/bin/bash", "-c", "uv pip install --strict " + project + " .[test]"})
    }
    buildContainer = buildContainer.WithExec([]string{"/bin/bash", "-c", "df -h /dev/shm; pytest -rP unit_tests.py "}, dagger.ContainerWithExecOpts{UseEntrypoint: true, InsecureRootCapabilities: true})
    return buildContainer

so /tmp/entrypoint.log logs every time that it is executed, and the terminal doesn't show any additiional entries

[edgy cliff]

I'd assume temrinal is not running the entrypoint.sh correct?

[manic socket]

right

[edgy cliff]

so you can't see the data in /dev/shm?

[manic socket]

let me put a tmp file in /cache

[edgy cliff]

so what happens if you manually run /entrypoint.sh once in the terminal?

that should re-configure the /dev/shm appropriately

since /cache should be set correctly IIRC

[manic socket]

oh yeah that works

it has three bind mounts somehow:

$ mount | grep shm
shm on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,relatime,size=65536k,inode64)
/dev/mapper/cl-docker--xfs on /dev/shm type xfs (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,usrquota,prjquota)
/dev/mapper/cl-docker--xfs on /dev/shm type xfs (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,usrquota,prjquota)
/dev/mapper/cl-docker--xfs on /dev/shm type xfs (rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,usrquota,prjquota)

[edgy cliff]

lol. Once's not enough 😄

[manic socket]

the nice thing about the entrypoint capturing to that log is that I can see the test output

[edgy cliff]

@manic socket so besides the 3 mount thing. Seems like you tricked it to make it work?

it'd be nice to make this shared /dev/shm thing more native. If you find the time to open an issue in dagger/dagger that'd be much appreciated

so hopefully some other users can upvote and push for that

[manic socket]

yeah probably will make the root hackery a bool option

yeah it'd be nice to have parity with more real world capabilities, I guess

[edgy cliff]

makes sense. It's not easy to enable those knobs when the premise of the product is to run everything in a sandbox 😛

but yes, I agree that it's a matter of the right design + meeting users where they are

[manic socket]

I doubt that there are many users who need tons of shared memory, I don't understand why this was done this way, tbh!

but thanks for the help

[edgy cliff]

you mean in your apps?

[manic socket]

yeah this repo that's a part of our product