Offline usage of dagger

[oblique mural]

I work in a offline lab environment that would benefit greatly from dagger.

is there a list of requirements for running dagger in an offline environment?

[pulsar robin]

If you already have the dagger engine downloaded locally it should just work.

Well, as long as you use images and parts of the api that is offline 😄

When the dagger first runs it downloads the dagger-engine binary to your os' temp dir. For example on MacOS ~/Library/Caches/dagger/dagger-engine-vx.x.x, you can also run it yourself, and set the required env variables and it should work from anywhere.

I don't know if there is an example for doing this though.

the variables are mentioned here in the python sdk: https://github.com/dagger/dagger/blob/4b55ecb740371e9ea6ddaac7e92a6518c1c3e352/sdk/python/src/dagger/engine/conn.py#L23-L34

When you start the dagger engine manually it will print these variables in a json response.

Library/Caches/dagger 
❯ ./dagger-0.3.13 session
{"port":53391,"session_token":"xxx"}

[ruby hemlock]

^ this.

Just make sure to pull images by digest and not by tag, otherwise dagger will try to resolve the tag and it'll fail if it can't

if your environment is entirely offline, the tricky part will be populating the build cache so your pipelines have all the required images cached for them to run. We currently don't have any instructions on how to do that since it hasn't been something many users requested

happy to provide assistance if you want to pursue that path @oblique mural.

[pulsar robin]

I personally do something sorta similar, I just rely on a local harbour instance for all my image needs. I just pull a set of images once a week 😄 That may or may not fit your requirements. Mainly because renovate will exhaust my api budget towards docker.io

[cunning thicket]

I guess it would be most easy to run a local registry with a set of images that you are going to use.

[oblique mural]

i guess i'm just not really sure what dagger is expecting. i do alot of offline work, but this would need to scale. I guess i may be looking into deploying harbor for this use case.

ideally this would "Just work" locally for my team of devs AND in our gitlab pipelines

I'll do some more investigating and see what i come up with.

[oblique mural]

Is what you're suggesting simply a registry with the images contained in it? would configuring my container daemon to use a mirror be sufficient?

[ruby hemlock]

Yes, that should be sufficient

[oblique mural]

for my follow up question lol

is there a list of images that would be pulled and does dagger have high churn with those images?

the container images i would need for development, i already have availible. I'm more concerned with what dagger needs to operate

[ruby hemlock]

dagger only pulls the engine image. As long as you don't bump your SDK, no new image will be pulled

and we're currently keeping released stable engine images undefinitely for now

[oblique mural]

cool awesome! i noticed that it also tried to make a call to dagger.io to pull some digest?

again, very novice. only just started messing with it a few days ago.

if at anytime there are docs that would answer these majority of these questions, feel free to point me at them.

[oblique mural]

So I have one issue. I’m in an offline environment and dagger seems to be wanting to pull a checksum.txt file that is not on my network. Is there a workaround for that?

[ruby hemlock]

checksum.txt? What are you exactly trying?

[oblique mural]

So I’m offline. I have moved the image over and the sdk. Everything builds just fine but I guess when it tries to reach out to the engine, it says “woah I can’t find some checksum.txt file” to validate.

The image IS there and loaded into docker.

c := dagger.connect() fails

I just stepped away from my machine but I’ll try to get the exact error message. I would suggest to replicate, create a vm with docker. Move the image and a dummy project over, turn of internet, and try to build your project.

[ruby hemlock]

this is what I did the other day that we were talking about it and it worked in my machine

I basically built a pipeline, disabled internet, and was able to successfully re-build it

[solemn bay]

The checksum is for downloading the CLI. You can download it yourself and set _EXPERIMENTAL_DAGGER_CLI_BIN to its path. Then the CLI will attempt to pull the engine container image. You’ll need to run it yourself as well and set _EXPERIMENTAL_DAGGER_HOST. Like Marcos says, the first time you run something this will be done for you. The next time should pick up that you already have these things.

[oblique mural]

Ahhh I see now. I’ll just download it via the instructions and see if that fixes things thank you so much. I think I see better now how the pieces of dagger fit together.

[oblique mural]

OK so what you're saying is that you had an online machine, built, turned off internet, and then it worked because those items were cached on the host.

This would be fine under most circumstances, however, to leverage dagger in a meaningful way for my use case, I am eventually going to have to distribute the artifacts needed to run dagger in an offline environment.

I downloaded the CLI and set the experimental cli bin and the cli works fine. I'm past the checksum issue, but It is still trying to pull the docker image even though it is present on the host machine. this is because it is calling out to registry.dagger.io, which does not exist on my network.

If this was an online environment. I would use dagger like crazy, but the limitations i'm running into are making this less and less viable for me.

[ruby hemlock]

I think it should still work. There's probably something that we're missing here. Let me do some more testing and come back to you

[solemn bay]

Did you try setting _EXPERIMENTAL_DAGGER_HOST like I said?

[oblique mural]

I did. What value is that supposed to be set to? I have it set to an internal registry. I’ve retagged the image and pushed of course. It still tries to pull from registry.dagger.IO

[solemn bay]

Sorry I was on my phone. It's _EXPERIMENTAL_DAGGER_RUNNER_HOST=docker-container://<name-of-running-engine-container>.

[ruby hemlock]

@oblique mural I just did a test completely from scratch by wiping the buildkit cache, setting up a local registry and without setting any env variable and it works for me with the v0.3.13 engine. This is the steps that I followed

1. import/pull registry.dagger.io/engine:v0.3.13 and copy the ~/.cache/dagger/dagger-0.3.13 file in your machine to your server
2. run a local registry with docker run -d -p 5000:5000 --rm --name registry registry:2
3. import/push an image to your local registry
4. Disable networking entirely
5. start your engine with the following command docker run --privileged -v $(pwd)/engine.toml:/etc/dagger/engine.toml -d --name dagger-engine-57eff2cd90629414 registry.dagger.io/engine:v0.3.13. In my case engine.toml so my engine can access the registry I started insecurely. Here's how that file looks like:

[registry."172.17.0.1:5000"]
  http = true
  insecure = true

^ you'll need to modify that IP depending on the ip your env configuration

5. Run the pipeline below

here's the simple pipeline I was testing with:

package main

import (
    "context"
    "fmt"
    "os"

    "dagger.io/dagger"
)

func main() {
    ctx := context.Background()
    c, _ := dagger.Connect(ctx, dagger.WithLogOutput(os.Stderr))

    out, err := c.Container().From("172.17.0.1:5000/alpine").WithExec([]string{"echo", "hello"}).Stdout(ctx)
    if err != nil {
        panic(err)
    }

    fmt.Println(out)
}

[ruby hemlock]

This pipeline executed successfully offline. I got a warning message when the pipeline started about not being able to resolve registry.dagger.io but it didn't fail:

 go run main.go 
failed to resolve image digest: Get "https://registry.dagger.io/v2/": dial tcp: lookup registry.dagger.io on 127.0.0.53:53: server misbehaving
falling back to leftover engine
#1 resolve image config for 172.17.0.1:5000/alpine:latest
#1 DONE 0.0s

#2 importing cache manifest from dagger:10686922502337221602
#2 DONE 0.0s

#3 from 172.17.0.1:5000/alpine
#3 resolve 172.17.0.1:5000/alpine:latest done
#3 sha256:63b65145d645c1250c391b2d16ebe53b3747c295ca8ba2fcb6b0cf064a4dc21c 3.37MB / 3.37MB 0.0s done
#3 extracting sha256:63b65145d645c1250c391b2d16ebe53b3747c295ca8ba2fcb6b0cf064a4dc21c 0.0s done
hello

LMK if / what doesn't work for you and we can troubleshoot together 💪

[oblique mural]

so i haven't gotten around to testing. i may later today or at the latest tomorrow.

The biggest problem here for me is that despite working in an "offline" environment. I have an extensive infrastructure i am responsible for. If i'm going to commit this to my code repo for usage. I can gladly stand up one or 2 pieces of static infrastructure if needed, but it would be untenable and unreasonable to completely reconfigure and reproduce these steps on all of my runners (100+ that a dagger job could land on) and other bits of internal infra to make it compatible with this tool.

Being able to pass in some arguments via environment variable would be nice.

I searched the go sdk and i found a reference to _EXPERIMENTAL_DAGGER_CLI_BIN but not to _EXPERIMENTAL_DAGGER_HOST

If i need to make an issue for tracking I can. It would be incredibly ideal for me to be able to set these values and have things "Just work™️"

A reference to _EXPERIMENTAL_DAGGER_RUNNER_HOST is not found either.

I don't think the problem i'm ultimately trying to solve is getting it working on my machine offline. The problem is how to proliferate the needed artifacts across a large offline environment without having to create artisanal hand crafted servers specifically designed to run dagger.

... and having it work of course 😆

[ruby hemlock]

@oblique mural the only thing you need to do is to place the dagger CLI in a specific folder and have the correct engine image present in your docker daemon, that's all. I'm not sure what you mean by "but it would be untenable and unreasonable to completely reconfigure and reproduce these steps on all of my runners".

The example that I posted above is a very specific one that starts a registry to show you that offline work is actually possible, you don't need to execute all those steps

[oblique mural]

ok cool that is pretty concise. placing a single cli tool and docker image on machines should be ok. I did get hung up on the fact that it needed the cli as well.

[ruby hemlock]

here's more information about it https://github.com/dagger/dagger/blob/main/core/docs/d7yxc-operator_manual.md

[oblique mural]

this is a really nice doc

immediately answer some questions i have.

are the sdk's really just... wrapping a graphQL api?

i know extremely little of graphql but it's really cool what you all have created with it.

[ruby hemlock]

pretty much, yes. With some custom logic for some other stuff, but yeah..

[oblique mural]

reading that doc, i still can't find a _EXPERIMENTAL_DAGGER_RUNNER_HOST reference. is this something that the CLI will look for as opposed to my go program?

nevermind. I think i'm tracking it down through the docs.

[ruby hemlock]

yes, our CLI uses that env variable to lookup where the dagger daemon is. The main reason why this is not documented in detail is because as you can see, it's still experimental since this interface is likely to change in the future

[oblique mural]

So i managed to get it working but i don’t think the juice is worth the squeeze for my offline environment yet

I’ll be keeping an eye on dagger and looking for an offline installation option.

[ruby hemlock]

@oblique mural 👍 is there anything else you had to do besides the two steps we previously discussed? Just trying to understand where the pain points are so we can map them for future improvements

[oblique mural]

no.

ideally I need a better way to distribute the engine image without renaming a ton of variables. perhaps passing the shasum of the image and the image name for the dagger engine, idk.

if i was one guy doing a thing on one offline computer, this wouldn't be a massive issue, but ideally, I would be able to get this working on my existing offline infrastructure without having to reconfigure a ton of things on each individual host, setting environment variables, launching the container, ensuring that root ca for my environment is installed in the container or the daemon is configured for insecure registries is just too much

ideally being able to pass an option that specifies where the cli tool and container image lives in the sdk would be amazing

i mean, you could create a cli container that has nothing but the cli binary in it and have the sdk copy that out of the container to an appropriate location maybe?

[ruby hemlock]

Got it, thx 🙏 ! Well.. hopefully the community starts asking more about the use-case it so it can get enough visibility to bump it in the priority queue

[solemn bay]

@oblique mural you can automate all of that without much effort I think. You can automate packaging it up in a tarball (handy for updating), send it to your offline machines, and run the unpacking automation locally to setup the engine. Updating only needs to run once. Then it’s just a matter of distributing the new package to the machines and running the unpacking script. I’m interested in making a POC for this, just don’t have much bandwidth for now.

[oblique mural]

Yeah and I’m capable of doing that via ansible, but it is adding complexity into an already quite complex workflow. My use case is unusual. I’m responsible for a chunk of infra. I can do what I want with it but the end goal would be greater adoption in the company. If I came to them with a mage file that invoked the pipeline without issue, they would absolutely cream (sorry for being crass 😆) but to caveat this with “well you have to build the image with our certs and make sure it’s available and running on every one of your runners, install the cli, and set these environment variables, im not going to get much traction. It’s like having a large kubernetes cluster but you have to manually push your images to each host that “may” run your pod.

If I get time, I may make a pull request as a crappy POC. I’m not a dev but I poked around last night and might be able to whip something up.

You all have made a great product and I get that some of the design considerations are made to protect your equity in it.

Is there a patreon I can donate to?

[ruby hemlock]

thx for the kind words @oblique mural ! We're not taking donations ATM, but it always helps us to spread the word about the product and provide feedback like the one you shared to help us improve it 💪