#Is it though???
glacial nest
I'm only creating a custom dagger engine in order to provide the corporate custom CA certificate (SSL filtering).
$ docker run \
--volume /var/lib/dagger \
--volume $HOME/.cache/ca-certificates:/usr/local/share/ca-certificates \
--name dagger-engine-custom \
--privileged \
--detach \
--restart always \
registry.dagger.io/engine:v0.16.3
delicate bluff
that seems ok. Looks like OrbStack might not be doing the size allocation appropriately
is there a way you can tune that somehow?
I'm not an Orbstack user here
glacial nest
I had a look at the OrbStack docs but couldn't fine anything.
Are you a macOS user?
delicate bluff
nope, linux here 😭
glacial nest
Fair enough, lucky man.
Docker Desktop for macOS performance is terrible when compared with OrbStack.
And you really feel it when you're running on a 16GB MBP. 😢
delicate bluff
it's very strange that the dagger-custom-engine container shows so little disk though
glacial nest
Given the output of this command I think I'll have to put it down to a bug in OrbStack, looks like some sort of overflow.
$ docker system df
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 1 1 485.2MB -2.94e+08B (-60%)
Containers 1 1 507.9kB 0B (0%)
Local Volumes 4 1 26.79GB 4.357GB (16%)
Build Cache 0 0 0B 0B
@glacial nest if you run a dagger -c "container | from alpine | terminal" . What space do you see there with df -h?
glacial nest
$ dagger -c "container | from alpine | terminal"
● Attaching terminal:
container: Container!
Container.from(address: "docker.io/library/alpine:latest@sha256:a8560b36e8b8210634f77d9f7f9efd7ffa463e380b75e2e74aff4511df3ef88c"): Container!
dagger / $ df -h
Filesystem Size Used Available Use% Mounted on
overlay 282.7G 7.9G 274.8G 3% /
overlay 282.7G 7.9G 274.8G 3% /etc/resolv.conf
/dev/vdb1 282.7G 7.9G 274.8G 3% /etc/hosts
overlay 282.7G 7.9G 274.8G 3% /.init
tmpfs 64.0M 0 64.0M 0% /dev
shm 64.0M 0 64.0M 0% /dev/shm
tmpfs 64.0M 0 64.0M 0% /proc/keys
tmpfs 64.0M 0 64.0M 0% /proc/timer_list
tmpfs 3.9G 0 3.9G 0% /sys/firmware
that sounds correct. Curious why the k3s container gets so little space though 🤔
glacial nest
Yeah it's odd.
Maybe the issue is with my k3s.yaml config?
.withExec(['sh', '-c', `
cat <<EOF > /etc/rancher/k3s/config.yaml
kubelet-arg:
- "eviction-hard=memory.available<500Mi,nodefs.available<75%"
node-label:
- "topology.kubernetes.io/zone=k3s"
- "region=primary"
EOF
`
I don't think so. I see a similar thing locally. I'm investigating
glacial nest
I added that kubelet-arg because I got errors like this before.
The node was low on resource: ephemeral-storage.
Only needs this part thought it seems.
kubelet-arg:
- "eviction-hard=memory.available<500Mi
I think it migh be related to the cgroups nesting entrypoint. Checking that
no, that's not it
keep looking 👀
delicate bluff
glacial nest
@delicate bluff great thanks
delicate bluff
<@336241811179962368> great thanks
regal hawk
Docker Desktop for macOS performance is terrible when compared with OrbStack.
A...
Is orbstack that much better? I ought to try it then. Have you tested emulation with it? I haven't had luck getting proper emulation to work with podman so I'm sticking to docker desktop.
For example, trying to run withExec inside a linux/amd64 image (to install tools for publishing) fails on my machine (M1 mac)
glacial nest
@regal hawk both OrbStack and Docker Desktop use Rosetta in order to achieve linux/amd64 container emulation
Give it a shot, https://docs.orbstack.dev/settings#use-rosetta-to-run-intel-code
regal hawk
I wonder what podman is doing then
glacial nest
regal hawk
I will give this a try. What differences are you seeing with orbstack over docker desktop? Just speed I assume?
regal hawk
https://podman-desktop.io/docs/podman/rosetta
still fails to do emulation though. It's something to do with security
regal hawk
the claims on their homepage are founded
good to know! Let me try
glacial nest
regal hawk
that's incredible! must make a drastic difference for dagger.
orbstack is not even mentioned in dagger docs. CC @stiff citrus https://docs.dagger.io/ci/integrations
glacial nest
I mean they're all just implementations of the same thing
regal hawk
you would think.
glacial nest
lol
Most of the magic and differences come from how they virtualize a Linux host and integrate the host OS with it
Which is why if you're fortunate to run on straight Linux you dodge most of the short comings
regal hawk
indeed.
docker has their own VMM now too. I didn't notice a performance difference though
delicate bluff
checking @glacial nest
@glacial nest seems to be ok here:
just called the module in a brand new engine and not getting any errors
have you tried changing the cluster name just to make sure it'll use a new cache volume?
delicate bluff
@glacial nest is there a chance you could connect your dagger to cloud and send a trace?
that'd be easier to check from our side
stiff citrus
@delicate bluff @regal hawk is there anything specific to orbstack that needs an integration page? I'm not on mac but it looks like you just download and run it, presumably dagger will then work ootb?
glacial nest
<@336241811179962368> <@163822683799158784> is there anything specific to orbst...
Yes it works ootb, which is why I don't think an integration page entry is necessary
@delicate bluff I can't run dagger -m github.com/marcosnils/daggerverse/k3s@v0.1.8 call --name "test" server up on it's own sadly because I need to provide the custom CA cert to k3s by mounting the cert file at /etc/ssl/certs .
I need to do like with-container , with-mounted-file or something
I'll try and work out the command
delicate bluff
Yes it works ootb, which is why I don't think an integration page entry is neces...
np! if you can connect to dagger cloud and send me the trace that also works 🙏
regal hawk
<@336241811179962368> <@163822683799158784> is there anything specific to orbst...
I will also do that setup and let you know. IMO, orbstack at least deserves an honorable mention since both nerdctl and podman are mentioned. I don't know if it necessarily needs to be in the integration section. That's where the other two are so I linked that page.
oh... now I remember why I am not using orbstack - https://docs.orbstack.dev/licensing. We don't have a license 🙂
glacial nest
oh... now I remember why I am not using orbstack - https://docs.orbstack.dev/lic...
They do offer a 30 day trial of Pro for commercial use.
https://docs.orbstack.dev/licensing#trial
I ended up buying a license, for testing purposes.
regal hawk
they prohibit personal licenses in a commercial setting. I can use it for the trial sure, but won't be able to use it afterwards.
glacial nest
they prohibit personal licenses in a commercial setting. I can use it for the tr...
For what it's work Docker Desktop also requires a 'license' after a company meets certain requirements.
https://docs.docker.com/subscription/desktop-license/
A free alternative might be something like https://lima-vm.io/
I haven't benchmarked that one in awhile though.
regal hawk
For what it's work Docker Desktop also requires a 'license' after a company meet...
Well aware 🙂 We do have a DD license. Easier to get a company to buy into DD than something else. We were already well invested in Docker anyway
glacial nest
Well aware 🙂 We do have a DD license. Easier to get a company to buy into DD th...
I'm literally in that same boat
regal hawk
A free alternative might be something like https://lima-vm.io/
I haven't benchm...
I've been testing lima and it works well.
glacial nest
@delicate bluff for some reason my method of mounting my custom CA cert at /etc/ssl/certs as part of the custom k3s container setup has stopped working. 🤷♂️
Now I'm getting errors like this, when trying to server up.
E0324 17:06:49.463489 16 kuberuntime_manager.go:1237] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to start sandbox \"f33c6044d0257a8b6266b7bea899badef86ac23b240943370c88ef7d54582dd8\": failed to get sandbox image \"rancher/mirrored-pause:3.6\": failed to pull image \"rancher/mirrored-pause:3.6\": failed to pull and unpack image \"docker.io/rancher/mirrored-pause:3.6\": failed to resolve reference \"docker.io/rancher/mirrored-pause:3.6\": failed to do request: Head \"https://registry-1.docker.io/v2/rancher/mirrored-pause/manifests/3.6\": tls: failed to verify certificate: x509: certificate signed by unknown authority" pod="kube-system/coredns-ff8999cc5-nldxx"
E0324 17:06:49.463650 16 pod_workers.go:1301] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"coredns-ff8999cc5-nldxx_kube-system(5a61c14e-3368-429d-a8fb-2b0953689d2b)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"coredns-ff8999cc5-nldxx_kube-system(5a61c14e-3368-429d-a8fb-2b0953689d2b)\\\": rpc error: code = Unknown desc = failed to start sandbox \\\"f33c6044d0257a8b6266b7bea899badef86ac23b240943370c88ef7d54582dd8\\\": failed to get sandbox image \\\"rancher/mirrored-pause:3.6\\\": failed to pull image \\\"rancher/mirrored-pause:3.6\\\": failed to pull and unpack image \\\"docker.io/rancher/mirrored-pause:3.6\\\": failed to resolve reference \\\"docker.io/rancher/mirrored-pause:3.6\\\": failed to do request: Head \\\"https://registry-1.docker.io/v2/rancher/mirrored-pause/manifests/3.6\\\": tls: failed to verify certificate: x509: certificate signed by unknown authority\"" pod="kube-system/coredns-ff8999cc5-nldxx" podUID="5a61c14e-3368-429d-a8fb-2b0953689d2b"
Do you have any idea how you're supposed to add a custom CA certificate so k3s can trusted the SSL filtered traffic within a corporate VPN?
delicate bluff
<@336241811179962368> for some reason my method of mounting my custom CA cert at...
@glacial nest I guess you need both the k3s cluster itself as well as the pods it creates to have access right?
or you're injecting the SSL certs to the pods in some other way?
I'm mentioing this because the dagger engine already has a predefined way to configure it to use SSL certs: https://docs.dagger.io/configuration/custom-ca/
delicate bluff
I'm mentioing this because the dagger engine already has a predefined way to con...
this adds whatever SSL cert you have to the dagger engine + all the containers / services it creates
glacial nest
Yeah but I think the rancher/k3s image is based on scratch which is why it's not done automatically
Hence why dagger -m github.com/marcosnils/daggerverse/k3s@v0.1.8 call --name "test" server up fails
Running on my dagger-engine-custom
delicate bluff
Yeah but I think the `rancher/k3s` image is based on `scratch` which is why it's...
right.. if that's the case, the engine won't provision the cert to that container 😬
