k3s not starting

[violet perch]

Hi again, i'm trying to follow this guide to run k3s for my integration tests that are deployed via helm https://github.com/dagger/dagger/issues/5292
I know that i have to adjust the code to work with Container.asService but i'll get to that later. For now this basic snippet fails:

import sys
import anyio
import dagger

async def main():
    async with dagger.Connection(dagger.Config(log_output=sys.stderr)) as client:

        out = (
            client.container()
            .from_("rancher/k3s")
            .with_entrypoint(["sh", "-c"])
            .with_exec(["k3s server --snapshotter native"], insecure_root_capabilities=True)
        )

        k3s = await out.stdout()

    print(k3s)

anyio.run(main)

Getting errors:

node_container_manager_linux.go:61] "Failed to create cgroup" err="cannot enter cgroupv2 \"/sys/fs/cgroup/kubepods\" with domain controllers -- it is in an invalid state" cgroupName=[kubepods]                                     
kubelet.go:1466] "Failed to start ContainerManager" err="cannot enter cgroupv2 \"/sys/fs/cgroup/kubepods\" with domain controllers -- it is in an invalid state"      

Using all the latest versions.

[heavy shard]

👋 what system are you currently running this on?

are you on linux? mac?

[heavy shard]

seems to be also happening with the latest version of the k3s image. Seems like something changed that that doesn't seem to be working with Insecure mode in buildkitl. Not sure if @teal perch might have a hint about what could be happening

doesn't seem to happen with docker run --privileged

[teal perch]

Don't have a ton of time to dive into this atm unfortunately but I did see that upstream modified the cgroup initialization script: https://github.com/moby/buildkit/pull/4582

So wondering if we want similar adjustments to ours https://github.com/sipsma/dagger/blob/43868b39fa745f894e4e331d7c00dff145726796/internal/mage/util/engine.go#L48-L48

Disclaimer: very superficial guess, but worth a try if anyone has time (cc @novel charm if you know anything else)

[heavy shard]

thx for sharing 🙌 trying this really quick

[heavy shard]

just updated the script really quick and doesn't seem to be the case. Seems like this requires a bit more 👀

[heavy shard]

ok, found the issue, not sure why/how this was working before

seems like k3s performs cgroup nesting (same script the dind image uses) "only" when it's executed as pid 1. Since we have our shim that's currently preventing that, the cgroup evacuation doesn't happen an it fails when trying to setup the cgroups kubernetes

a workaround is to run the cgroup nesting entrypoint oursleves. Will share a snippet in a bit cc @teal perch

^ that works. I'll update the code in the original issue

[heavy shard]

done, original issue updated https://github.com/dagger/dagger/issues/5292#issuecomment-1593750070

[novel charm]

😢 i suppose this is reasonable behavior, but this does seem bizarre to me - there's no really consistent way to see if you're inside a container, so checking pid 1 works i guess.

if we have some better ideas for how to detect this, we could potentially upstream a fix for this to k3s?

(not that i have any )