how to make dagger.Directory ?

[stuck bay]

hey team, I am VERY NEW to dagger (like 30mins in it), outside of command line, how to programmatically make dagger.Directory? just going through SDK / docs... I want dynamically pass params to .with_directory("/src", source, exclude=[".venv", ".ruff_cache"]) ...and that "source" is dagger.Directory, where mine is just str.

Thanks

[frail sleet]

👋 are you looking to write a dagger function that creates a directory?

[stuck bay]

no, I have a local dir... just want to convert it to object

[frail sleet]

OK but then what do you want to do with that object? Just query it with the dagger API, from the command-line?

[stuck bay]

from dagger import dag, function, object_type


@object_type
class RayInference:
    @function
    def build_env(self, source: dagger.Directory) -> dagger.Container:
        return (
            dag.container()
            .from_("python:3.11.9-bullseye")
            .with_exec(["pip", "uninstall", "-y", "virtualenv"])
            .with_exec(
                [
                    "curl",
                    "-sSL",
                    "https://install.python-poetry.org",
                    "-o",
                    "install-poetry.py",
                ]
            )
            .with_exec(["python3", "install-poetry.py"])
            .with_env_variable("PATH", "/root/.local/bin:$PATH")
            .with_exec(["poetry", "config", "virtualenvs.create", "false"])
            .with_directory("/src", source, exclude=[".venv", ".ruff_cache"])
            .with_workdir("/src")
            .with_mounted_cache("/root/.cache/pypoetry", dag.cache_volume("pypoetry"))
            .with_exec(["poetry", "install", "--no-root", "--without", "dev", "-vvv"])
            .with_entrypoint(["python", "/src/main.py"])
        )

    @function
    def build_all_services(self) -> list[dagger.Container]:
        service_paths = ["services/cdc-sink", "services/ingest"]
        service_dirs = [DIRECTORY(path) for path in service_paths]
        return [self.build_env(service_dir) for service_dir in service_dirs]

something like matrix build

I have N services locally, that I can glob

[frail sleet]

OK, so you're trying to call your build_env function from the command-line, and pass it a local directory as its source argument?

Oh I see, it's this line you're having trouble with?

    service_dirs = [DIRECTORY(path) for path in service_paths]

It looks like your build_all_services functions should itself receive either an array of directories, or a single directory containing all the services as subdirectories

Option 1:

    @function
    def build_all_services(self, service_dirs: list[dagger.Directory]) -> list[dagger.Container]:
        return [self.build_env(service_dir) for service_dir in service_dirs]

[stuck bay]

so, can I make dagger.Directory outside of cli? 🙂

the whole point here was to remove just a ton of sh scripts to get "services" in monorepo

[frail sleet]

Option 2:

   @function
    def build_all_services(
      self,
      services: Annotated[dagger.Directory, DefaultPath("./services")],
      ) -> list[dagger.Container]:
        service_dirs = [services.directory(name) for name in services.entries()]
        return [self.build_env(service_dir) for service_dir in service_dirs]

You'll probably like option 2 because it takes advantage of context directory access. Assuming your service dirs are at ./services at the root of your module (next to dagger.json then the directory will be constructed automatically

Example usage of option 2:

$ ls
dagger.json
services
$ ls services/
csc-sink
ingest
$ dagger call build-all-services

[stuck bay]

option 3, run some container, return list of dirs?

if I want to do a little more logic then DefaultPath

[frail sleet]

If you want your function to have access to files in your project repo, DefaultPath is your only way. But you can call a more advanced function which uses DefaultPath

[stuck bay]

glob(), exclude some services, read their configs

[frail sleet]

You can also combine DefaultPath with Ignore to filter the contents of large directories before they are passed to you

What kind of processing do you need to do for your option 3?

[stuck bay]

ok, got it... thanks ... I think what would like to do ideally, because this should be orchestrated in github actions, I will make 2 functions

1. get services

2. build_env

github action is going to matrix output from get-services to build-env

so I actually don't need to convert/make dagger.Directory in this case

[frail sleet]

Having GHA do the matrix makes sense if you want to dispatch each build on a separate VM (since Dagger cannot orchestrate multiple worker machines as part of a single call).

[stuck bay]

yep

thanks @frail sleet

[frail sleet]

I recommend testing single-machine, sometimes the overhead of spawning multiple machines in GHA is not actuallt worth it

ie. sometimes you get the yaml complexity and it actually slows down your pipeline. Depends on the workload of course.

[stuck bay]

we run private runners, different story

can I ask you another question here? @frail sleet

we use Tilt, and I really like dagger approach with dag.container()... but I still need to keep Dockerfiles for tilt

is there any workaround? because it looks like we are going to have dagger and Dockerfiles for tilt at the end 😦

unless we will just use dagger dockerfile approach

¯_(ツ)_/¯

sync dagger and dockerfile is going to be nightmare

[frail sleet]

Dagger has native compat for Dockerfiles, you can do Directory.docker_build

For exactly this kind of situation 🙂

does tilt actually trigger the docker directly? Or is it docker build -> docker push -> tilt ?

[stuck bay]

so I need to keep dockerfiles

tilt actually builds docker / replaces images and also does magic with volumes auto reload

[frail sleet]

well, it depends on your compatibility constraints. If you need to stay perfectly in sync between your dagger and tilt builds, then it appears you need to keep the dockerfile for that build, yes

Purely from the Dagger point of view, it's probably better to replace your Dockerfile with code

But sometimes you have to compromise for compatibility 🙂

[stuck bay]

exactly

[stuck bay]

the problem here that I already have 2 dockerfiles, because for tilt I need way more tools for local

so we do have dockerfile.prod, dockerfile.local 🙂

.prod already migrated to dagger... but while I can do a NORMAL programming language now in dagger, I thought I can just replace difference with dockerfile.local and use it in tilt

the only piece missing is hot reload

this one https://docs.tilt.dev/tutorial/5-live-update.html

need really to dig into that implementation and see if I can just replace it with dagger

[frail sleet]

Dagger doesn't have live rebuild, but it is coming 🙂 cc @near vortex

[stuck bay]

oh yeah

is there a PR / issue that I can trackdown?

[frail sleet]

I was just looking for that 🙂

[round arrow]

this is the one I normally point to, is this what you have in mind Solomon?

https://github.com/dagger/dagger/issues/6990

[frail sleet]

Thank god @round arrow I was going crazy looking for this

[stuck bay]

nice 🙂 thanks for your time @frail sleet @round arrow

[frail sleet]

@stuck bay if you have time to +1 that issue and share a few notes on your use case, it would be super useful 🙏

[stuck bay]

https://github.com/dagger/dagger/issues/6990#issuecomment-2210648282

🙂

I am not a single one

would be amazing, if you will have something like .with_mounted("", sync([""]))

where sync is a list of files you really want to watch

tilt has amazing API for that, even rerunning pip install in case lock file changes live_update=[ sync('./api/', '/app/api/'), run( 'pip install -r /app/requirements.txt', trigger=['./api/requirements.txt'] ) ]

[frail sleet]

The nice thing is that Dagger already has an API for filtering the files and directories you care about (via arguments, Ignore etc). And it already has an API for specifying which commands you want to run against which inputs 🙂

So all that is left to do, is make the above continuous on local file changes, and make it fast

[stuck bay]

just posted comment, again, thanks for the help 🙂 you are building amazing tool... I felt same, when I got into pulumi 4yrs ago 🙂

[frail sleet]

Thank you!

[random spade]

@stuck bay now that you're playing with Dagger I've been curious if / how Dagger could be a tilt, skafoold devspace, etc 10x better tool. If you happen to run into some ideas on what would be awesome, we're always looking for crazy stuff to prototype in our next hackathons 🙂

I have the feeling that theres' a big opportunity in the local dev loop space but as I'm not a tilt user myself, I'm probably missing the biggest pain points / opportunities.

[stuck bay]

uh, Tilt has so many issues.... just look at dependencies management, there is none for anything else outside of "resources".

would love to join on this front, I am pretty sure dagger can be full e2e and I don't know why it shouldn't

[random spade]

https://docs.tilt.dev/resource_dependencies.html

The two issues at the end of that article summarize it quite well. We can definitely improve this with Dagger.

[stuck bay]

For local development, I need to spin up MetalLB and then deploy an application like NGINX. The NGINX deployment creates a LoadBalancer service, which requires MetalLB to already be operational in the cluster. In Tilt, I need to define a k8s_resource for MetalLB, but since NGINX is installed using Helm, I can't directly set a dependency on the MetalLB resource. This creates a risk of a race condition where MetalLB might not be ready when NGINX tries to create the LoadBalancer service.