python

[short crow]

@tropic glacier and all other <@&1113692108755308593> fans, you now have a space to talk about all things Python and Dagger 🙂

If you are just getting started, this is a great place to go: https://docs.dagger.io/developer-guide/python/

And great video to watch: https://youtu.be/Noh9cHLV318

[tropic glacier]

@short crow thank you, the video is really good.

[severe cargo]

@short crow thank you it is amazing

[junior girder]

Heads up pythonistas, I'm making uv the default installer in the Python runtime container (for modules). There's an escape hatch though if you need it.

[ionic hare]

https://tenor.com/view/high-five-topgun-need-for-speed-gif-13303958

[junior girder]

Also plan on changing the file on init to be src/main/__init__.py instead of src/main.py. See #6863.

Also adding support for direct install from a requirements.lock file for pinned dependencies.

And, enabling choosing a different python version.

Those are all done already, just need finishing touches.

[south geode]

Haven't used uv but any speed increases are appreciated 🙌

[junior girder]

Yeah, here’s a few quick stats.

Without uv:

• python -m venv → ≈2.08s
• pip install -r lock → ≈4s (cold)
• pip install -e sdk → ≈6.30s/3.56s (cold/warm)

With uv:

• uv venv → ≈0.03s
• uv pip install -r lock → ≈0.9s (cold)
• uv pip install -e sdk → ≈1.36s/0.76s (cold/warm)

Since uv is so efficient installing from cache, I no longer have to worry about setting up a cache volume for the virtual environment, in a way that’s unique for each module. And since the uv cache can be shared between multiple modules, the first one will speed up all the others.

[distant escarp]

Yeah, here’s a few quick stats.

[high gust]

i have a function on a module
async def do_something(self, some: dict[str, Container] | None = {}) -> None:
but when i install that module it translate to
async def do_something(self, some: Sequence[str] | None = None) -> None:
how can i make it right i need to pass a dict insted of a sequence/list/array?

[junior girder]

Ah, that's a bug. It should error. You're not the first one to want it though: https://github.com/dagger/dagger/issues/6138

[junior girder]

Btw, reminder not to have {} as a default since it's mutable. The effect of it is reduced in modules, depending if you call other functions from within the module.

And a union like str | Container isn't supported too.

[high gust]

any way around or any other datatype that allow me to set this key->value kind of type?

[junior girder]

Only supported union is with None.

Yeah, I have something in mind, let me make a POC.

[high gust]

got it, but just to doble check, theres i no available way to pass a key->value data type to module/functions, right?

[junior girder]

Yep. Simplest is a list[str] with ["key1=val1", "key2=val2"] format.

Alternative would be a:

@object_type
class Map:
  key: str
  value: str

Simple enough in-module, but when using the generated client, you need a function to construct map objects.

So that becomes more complicated than what's worth.

[high gust]

another questio, i the signature of build from container is
def build(self, context: "Directory", *, dockerfile: str | None = "Dockerfile", target: str | None = "", build_args: Sequence[BuildArg] | None = [], secrets: Sequence["Secret"] | None = [],) -> "Container":
why if i build a wrap function on my code

    async def build_container_image(self, dockerfile: str, context: Directory, target: str, build_args: Sequence[BuildArg] | None) -> Container:```
it give me the following error
```Error: query module objects: json: error calling MarshalJSON for type *dagger.Module: returned error 400 Bad Request: failed to get schema for module "btd": failed to create function "buildContainerImage": failed to find mod type for function "buildContainerImage" arg "buildArgs" type```

basically the signatures are some how similar

[junior girder]

We don't support scalars, inputs and enums in functions yet.

$ dagger call introspect inputs name
BuildArg
PipelineLabel
PortForward

$ dagger call introspect enums name
CacheSharingMode
ImageLayerCompression
ImageMediaTypes
ModuleSourceKind
NetworkProtocol
TypeDefKind

$ dagger call introspect scalars name | grep -v -e Boolean -e Float -e Int -e Void -e String
CacheVolumeID
ContainerID
CurrentModuleID
DirectoryID
EnvVariableID
FieldTypeDefID
FileID
FunctionArgID
FunctionCallArgValueID
FunctionCallID
FunctionID
GeneratedCodeID
GitModuleSourceID
GitRefID
GitRepositoryID
HostID
InputTypeDefID
JSON
LabelID
ListTypeDefID
LocalModuleSourceID
ModuleDependencyID
ModuleID
ModuleSourceID
ObjectTypeDefID
Platform
PortID
SecretID
ServiceID
SocketID
TerminalID
TypeDefID

[junior girder]

As promised: https://github.com/dagger/dagger/pull/6884 (shoutout to @tropic holly for bringing us uv!):

[tropic holly]

Amazing!

[high gust]

if i have a module which define the following type

class NamedSecret(Type):
    """Named Secret Artifact."""

    name: str
    """Name of the secret."""

    value: Secret
    """Secret to use."""```
how can i import and use that type in my main module? i mean i like to do something like
``` NamedSecret(name="mysecret", secret=Secret())```?

[sullen frost]

@junior girder Can you suggest the best way to annotate Secret type with a default of None? Didn't get a render in Daggerverse. My implementation or Daggerverse's? cc @crude frigate

I tried this
https://daggerverse.dev/mod/github.com/jpadams/daggerverse/fossa@e5995c00d49e248557300f2e70da10670f5fa4bf

https://github.com/jpadams/daggerverse/blob/e5995c00d49e248557300f2e70da10670f5fa4bf/fossa/dagger/src/main.py#L30-L34

My intent is for the fossa_token to be optional. If you don't provide it, I assume you don't need to upload to Fossa SaaS, so we'll use a fossa-clioption for local stdout output only (call --output).

[alpine oxide]

BTW, Hynek has a nice, short commentary on uv at:
https://youtu.be/_FdjW47Au30?si=rz7fzP_mIuksHlX7

and a recent Talk Python inteviews Charlie about uv in more detail:
https://www.youtube.com/live/g5RWwvzfs0I?si=u6LWaGoouEaE6Zwc

[timber hull]

UV is neat. I want to expirement a bit with rye too; my old intel mac is no-longer supported by homebrew so I need a different approach for managing Python environments etc.

But in a totally unscientific shootout, creating a clean venv and installing our in-house library plus deps was 25% faster with uv than pyvenv + pip

[junior girder]

Yeah, I've been using rye exclusively locally. But still jump around hatch, uv and .venv/bin/activate too though.

[meager grove]

Whoa been a while since I looked at dagger stuff and now it seems like my pipeline is totally not fitting the latest dagger structure? I am not a python expert, so I put a lot of time troubleshooting to get this terraform pipeline working, but now I feel like I have to rework it all to be a module. Does anyone have a good terraform/terragrunt pipeline example?

[meager grove]

This was my pipeline based on the old dagger documentation https://pastebin.com/pUt7JD5S

[junior girder]

I can take a closer look (though not now), but it doesn’t require a full rewrite. It’s the same SDK as before, just wrapped in a reusable “package”.

For example, you don’t need argparse and you don’t need to manage the connection. Code in main needs to go in a function and the prints won’t work. Host needs to be replaced with explicit function arguments.

[meager grove]

Thank you! Yes honestly I mean whatever is best practice I’d love all the help I can get. It was hard for me to even get that working since there weren’t many examples for me to build off of, but once I have a solid foundation that’s when I’m most comfortable tweaking and adding onto it haha.

[meager grove]

Interesting though, I guess there must be some way to change those prints to be a verbose or error type of dagger output? It would definitely be ideal to still have the terraform output printed

[junior girder]

Terraform

[high gust]

how to get the string representation of a Directory object?

[ripe pawn]

Anyone have success installing and using a python dagger dependency? I can't seem to execute functions from any dependencies

[junior girder]

The string representation of any Dagger type is their ID: await directory.id().

Dependencies

[steep widget]

Is there anything special I need to do in order to get IDE integration for dagger sdk docs in vs code?

After running dagger init ... and then dagger develop I only see things like the attached screenshot, where I expect the nice robust docs that you get with go.

[junior girder]

IDE integration

[south geode]

@junior girder Quick one - a module name that includes a - becomes an _ in python right? E.g. test-module installed as a dependency becomes dag.test_module()?

[junior girder]

Yep

[south geode]

So at present, since we can't call a module from the SDK (without it also being a module)... I'm testing main_module by... writing another module (test_module), installing main_module as a dependency, and writing some @functions that use dag.main_module().with_method(args).method(args). I don't think I'm missing a simpler way to demonstrate testing a module as a dependency at the minute?

[junior girder]

I'm not sure I fully understand this: "since we can't call a module from the SDK (without it also being a module)". Can you elaborate?

Oh, you want to test your module, but as a client (i.e., using dag)?

Yes, you can include a test submodule and dagger -m test use .

[south geode]

That results in the same thing though right, a second module that can use dag.main_module().methods()

[south geode]

Yep - in cases where a module is largely designed to be used as a dependency rather than dagger call ... my users are going to find examples of it in use as a dependency in a second (test) module

[junior girder]

I'm agreeing with you, just making sure you mean the same. 🙂

[high gust]

is this

                container
                .with_registry_auth(address="us-east1-docker.pkg.dev", username="oauth2accesstoken", secret=dag.set_secret(name="GCP_ACCESS_TOKEN", plaintext=os.environ.get("GCP_ACCESS_TOKEN")))
            )```
the right way to auth to gcp artifact registry? also the right way to get an env variable? for some reason is getting null from the env var and it exists

[junior girder]

Secret from env

[high gust]

running on GHA for few days all fine. today just got this errors. but not sure what they mean. help please

[junior girder]

That's due to a release yesterday of beartype 0.18.0. See #1224656335094943744 message. This has been fixed in https://github.com/dagger/dagger/pull/6998 but also, today beartype released a new version reverting that breaking change. So I think if you try again it should work.

[hollow wyvern]

Hey folks! Is this the "correct" wait to call two functions in parallel and display the output of each?

    @function
    def fmt_check(self, source: dagger.Directory) -> dagger.Container:
        return self.mounted_source(self.container_terraform, source).with_exec(
            ["fmt", "-check", "-recursive"]
        )

    @function
    def tflint(self, source: dagger.Directory) -> dagger.Container:
        return self.mounted_source(self.container_tflint, source).with_exec(
            ["--recursive"]
        )

    @function
    async def check(self, source: dagger.Directory) -> str:
        tflint = self.tflint(source).stdout()
        fmt = self.fmt_check(source).stdout()
        return await tflint + await fmt

[junior girder]

Structured concurrency

[autumn tulip]

Hi, I want to package and serve my FastAPI project as so: ```py
@object_type
class Ci:
@function
def serve(self, source: dagger.Directory) -> dagger.Service:
"""Create a service from the image"""
return self.package(source).as_service()

@function
def package(self, source: dagger.Directory) -> dagger.Container:
    """package image"""
    return (
        dag.container()
        .from_("python")
        .with_workdir("/workdir")
        .with_directory("/workdir", source)
        .with_exec(["pip", "install", "uvicorn", "fastapi", "pydantic"])
        .with_exec(["uvicorn", "main:app"])
        .with_exposed_port(8000)
    )

@function
def get(self, source: dagger.Directory) -> str:
    """GETs the serving container"""
    return (
        dag.container()
        .from_("alpine")
        .with_service_binding("www", self.serve(source))
        .with_exec(["wget", "-O-", "http://www:8000/openapi.json"])
        .stdout()
    )

everything seems to be working fine, but wget aint able to connect:

I've been mostly following this guide: https://docs.dagger.io/manuals/developer/python/328492/services

[ionic hare]

How do you run the dagger function?

oh I see the problem

[autumn tulip]

dagger call get --source=../src

[ionic hare]

you need to configure the server to listen on 0.0.0.0 instead of 127.0.0.1

[autumn tulip]

thank you Solomon 🙂

works now: ```py
@object_type
class Ci:
@function
def serve(self, source: dagger.Directory) -> dagger.Service:
"""Create a service from the image"""
return self.package(source).as_service()

@function
def package(self, source: dagger.Directory) -> dagger.Container:
    """package image"""
    return (
        dag.container()
        .from_("python")
        .with_workdir("/workdir")
        .with_directory("/workdir", source)
        .with_exec(["pip", "install", "uvicorn", "fastapi", "pydantic"])
        .with_exec(["uvicorn", "main:app", "--host", "0.0.0.0"])
        .with_exposed_port(8000)
    )

@function
def get(self, source: dagger.Directory) -> str:
    """GETs the serving container"""
    return (
        dag.container()
        .from_("alpine")
        .with_service_binding("www", self.serve(source))
        .with_exec(["wget", "-O-", "http://www:8000/openapi.json"])
        .stdout()
    )

[ionic hare]

Love it!

Are you going to use that for integration tests?

[autumn tulip]

yes

I will keep developing with dagger tho. I want to stretch it to the left (kinda try to use it as an application framework) and to the right (CD and actual hosting)

[lone kelp]

Hi, trying out dagger using the python-sdk. Works well on my laptop (M1) but when I run dagger init --sdk=python in an empty dir on a debian machine I get

Error: failed to generate code: input: moduleSource.withContextDirectory.withName.withSDK.withSourceSubpath.asModule resolve: failed to create module: select: failed to update codegen and runtime: failed to generate code: failed to call sdk module codegen: select: call function "Codegen": process "/runtime" did not complete successfully: exit code: 2

Stdout:
json: error calling MarshalJSON for type *dagger.GeneratedCode: input: container.from.withMountedCache.withExec.withEnvVariable.withEnvVariable.withMountedDirectory.withMountedDirectory.withExec.withNewFile.withExec.withMountedDirectory.withWorkdir.withExec.withExec.withExec.directory resolve: unlinkat /var/lib/dagger/runc-overlayfs/cachemounts/buildkit2850587130/runtime/template/src: invalid argument

Tried dagger init --sdk=go and that seems to work
Not sure how to troubleshoot.

[hollow wyvern]

Do I just run dagger develop --sdk=python with a new version of dagger to upgrade the sdk?

[ionic hare]

I love that 🙂

[junior girder]

Yes, but only needed for IDE integration.

Debian init

[plain current]

hi, there someone here who have a code about cloning a repo using dagger ?

[junior girder]

https://archive.docs.dagger.io/0.9/cookbook/#add-git-repository-as-directory-to-container

[plain current]

thx you

[steep widget]

I will keep developing with dagger tho.

[south geode]

With the new build backend and the required [project] in pyproject.toml I'd appreciate an example of using Poetry if you have one @junior girder ?

[junior girder]

Sure! https://github.com/helderco/daggerverse/blob/main/poetry/pyproject.toml 🙂

What's required depends on the build backend. If you don't specify one, it falls back to setuptools (legacy). I've added hatchling instead because it has better defaults, but it can be poetry, pdm, etc....

[south geode]

Ah I meant if the hatchling build backend can be combined with Poetry

Hatchling requires a [project] block, Poetry doesn't provide one and complains if you do

[junior girder]

No, makes no sense to do so 🙂

[south geode]

So for Poetry users the suggestion is to not use Hatchling

[junior girder]

When you build a package, there needs to be only one to rule the build 🙂

[junior girder]

Exactly. You can use some Hatch features, concurrently with Poetry, just not the build.

Btw, did you know Poetry is joining the standard? https://github.com/python-poetry/poetry/pull/9135

[south geode]

I haven't seen that, no

[junior girder]

It'll be easier to switch when that lands.

[mystic basalt]

Hello, thanks for the great work on Dagger! I'm wondering if there's a good way to transfer files from a container back to a local filesystem, similar to how volume mounting works with docker run -v src:dst ... commands. I saw with_mounted_directory https://dagger-io.readthedocs.io/en/sdk-python-v0.11.0/client.html#dagger.Container.with_mounted_directory but this seems to be one-way, adding the files to the container without syncing them back to the local system. Is there a way to do both (sync to the container and back out again)?

[steep widget]

Hello, thanks for the great work on

[junior girder]

Anyone has an example of a module managing Poetry for @median plover? I have a Poetry module but it's just for managing the module's dependencies, not for pipelines that use Poetry.

[median plover]

I am making progress with this guide thanks https://archive.docs.dagger.io/0.9/sdk/python/628797/get-started/

[ripe pawn]

Can seems to figure out how to create optional, module-wide attributes. Anyone know?

@object_type
class Tests:
    test_secret: dagger.Secret = field(default=dagger.Secret)

This is what I have, it I don't pass in test_secret, it throws an error

[junior girder]

Default secret

[ionic hare]

Isn't that the old quickstart? Is it still up-to-date @junior girder ?

[junior girder]

That’s right @median plover, that’s an older QuickStart. I think it’s because of #dotnet message but notice that I didn’t pull the latest docs for you there because you were looking into Dotnet which doesn’t have the modules support that the new docs are based on. If you’re working with a module supported SDK though (Python, Go, TypeScript), I recommend trying out the latest documentation.

[pure kettle]

I upgraded from dagger 0.10 -> 0.11 and I now get the following warnings at the end of my pipeline:

{"level": "WARNING", "time": "2024-04-19 12:43:30,280", "location": "opentelemetry.exporter.otlp.proto.grpc.exporter:_export:293", "message": Transient error StatusCode.UNAVAILABLE encountered while exporting traces to localhost:4317, retrying in 1s.}
{"level": "WARNING", "time": "2024-04-19 12:43:31,284", "location": "opentelemetry.exporter.otlp.proto.grpc.exporter:_export:293", "message": Transient error StatusCode.UNAVAILABLE encountered while exporting traces to localhost:4317, retrying in 2s.}
{"level": "WARNING", "time": "2024-04-19 12:43:33,290", "location": "opentelemetry.exporter.otlp.proto.grpc.exporter:_export:293", "message": Transient error StatusCode.UNAVAILABLE encountered while exporting traces to localhost:4317, retrying in 4s.}
{"level": "WARNING", "time": "2024-04-19 12:43:37,296", "location": "opentelemetry.exporter.otlp.proto.grpc.exporter:_export:293", "message": Transient error StatusCode.UNAVAILABLE encountered while exporting traces to localhost:4317, retrying in 8s.}
{"level": "WARNING", "time": "2024-04-19 12:43:45,301", "location": "opentelemetry.exporter.otlp.proto.grpc.exporter:_export:293", "message": Transient error StatusCode.UNAVAILABLE encountered while exporting traces to localhost:4317, retrying in 16s.}
{"level": "WARNING", "time": "2024-04-19 12:44:01,308", "location": "opentelemetry.exporter.otlp.proto.grpc.exporter:_export:293", "message": Transient error StatusCode.UNAVAILABLE encountered while exporting traces to localhost:4317, retrying in 32s.}

Has anyone else had this?

[junior girder]

Yes, sorry about that. This is currently being worked on, see if this works for you: https://github.com/dagger/dagger/discussions/7116#discussioncomment-9157926

[pure kettle]

Will try that, thanks!

That implementation removed the Transient error logs, but I still receive the "Already shutdown, dropping span" log

[junior girder]

Even with that logging statement?

[pure kettle]

I added the following lines (before async with dagger.connection()):

logging.getLogger("opentelemetry").addHandler(logging.NullHandler())
opentelemetry.trace.get_tracer_provider().shutdown()

[junior girder]

Does your script configure logging?

[pure kettle]

Yes

I added those lines to my log-configuration function

[junior girder]

Ok, it may need adjusting if you already configure logging.

[pure kettle]

This is my configuration:

log_format = (
    (
        '{"level": "%(levelname)s", '
        '"time": "%(asctime)s", '
        '"location": "%(name)s:%(funcName)s:%(lineno)s", '
        '"message": %(message)s}'
    )
    if get_flag_from_env("INCLUDE_LOG_LOCATION", default=False)
    else (
        '{"level": "%(levelname)s", '
        '"time": "%(asctime)s", '
        '"message": %(message)s}'
    )
)
logging.basicConfig(
    format=log_format,
    level=level,
    force=True,
    handlers=[
        logging.StreamHandler(sys.stdout),
    ],
)
# remove httpx logs (INFO logs every time a request is made - annoying):
logging.getLogger("httpx").setLevel("WARNING")
# fix opentelemetry logs issue: https://github.com/dagger/dagger/discussions/7116#discussioncomment-9157926
logging.getLogger("opentelemetry").addHandler(logging.NullHandler())
opentelemetry.trace.get_tracer_provider().shutdown()

I could always just add logging.getLogger("opentelemetry").setLevel("ERROR"), but that seems like a bit of a bad hack

[junior girder]

Convo continued in #1230840665215336511 message.

[alpine oxide]

Same here, @autumn tulip . I'm a huge python fanboy and happy to pair up on python dagger projects.

[autumn tulip]

awesome!

[stark dagger]

👋 just came across an unwanted behavior which I'm not sure if it has been raised before. While working in my dagger python module, if I initialize a virtual environment in it so I can get code completion in my IDE, every time I run dagger call, my whole .venv directory will get uploaded to the execution context. In many cases this ends up making my dev loop very slow as generally the .venv directory tends to grow quite fast

cc @junior girder

[junior girder]

This is well known. Notice the TIP in IDE Integration and issue Add to exclude by default with dagger init.

If you use poetry or hatch, these tools create the virtual env in a global location so it doesn't affect this. But with rye, uv and bare pip it's more common to put it in a local .venv.

[stark dagger]

oh! totally missed that. I also didn't remember about the global include/exclude thing in dagger.json. Thx 🙏

[prime cloak]

Hi,
am realy new on dagger and because its another type of thinking, i will need some help.
Am trying to find some project/rs, or git repos which using trivy scanner on helm, docker images, teraforms, with gitlab pipelines...
The new and old documentation makes me realy out of context...
am using now only codegen

1. where to define dockerfile or is some plugin to program him in the ... /trivy/dagger/src/main/init.py
2. main/init.py have same function like main.py?
3. where to define ci-piplines.yaml, dockerfile, taskfile...
4. Is somewhere some good functional project from which i may look on code and find the best ways how to working with

[ionic hare]

Lunch with @tropic holly 😁 of ruff & uv fame

[median plover]

it took me a month to find a working dagger example in python 😓 anyone have any other good reference examples I could learn from? https://github.com/samalba/hf-model-ops/tree/main/ci

[steep widget]

This is one I am proud of recently, includes parallelism, happy to answer any questions you may have

GitHub Actions Config: https://github.com/levlaz/boundary-layer/blob/master/.github/workflows/python-dagger.yml

Dagger Portion: https://github.com/levlaz/boundary-layer/blob/master/dagger/src/main/__init__.py

[median plover]

starred! many thanks!

airflow dags look interesting too

not only looks interesting it looks like I have to try this.

[worldly island]

👋 folks,

I had some fun in the past couple of days writing a dagger module for Elasticsearch in python 🙂

Great work. Love the dev experience!

I tried to follow the best practices by checking the doc and other python modules examples.

Looking for feedback as I am pretty sure there is a lot of room for improvements.

https://daggerverse.dev/mod/github.com/mgreau/daggerverse/elasticsearch@f444fd7f778cc017cf5ac052aac6d84cf32aedfa

cc @ionic hare

[prime cloak]

Hi all,
how to setup dagger to have socket like in this command (-v /var/run/docker.sock:/var/run/docker.sock):
docker run --network=host -v /var/run/docker.sock:/var/run/docker.sock aquasec/trivy image gitlabregistry.devct.cz/microservice/taxonomy-api:1.6.6
how to setup --network=host

[south geode]

Hi all,

[junior girder]

elasticsearch :: Daggerverse

[alpine oxide]

Thanks to a series of fortunate events, I WILL be at PyCon this week!
If any of you are there, please say hello.

[unkempt rune]

I successfully created a pyhon pipeline in the file __init__.py as in the docs.
Now I want to move this code in a new file names main.py since I think that it not belongs in __init__.py

I did and now when I run "dagger call my-function..." I got the error:

Module “cicd” doesn't define any top-level functions or a “Cicd” class decorated with @object_type.

(I named the class "Cicd")

What did I miss ?

[junior girder]

Why don't you think the pipeline belongs in __init__.py? 🙂

Did you put it in src/main/main.py, or src/main.py?

You can read more about this in Develop with Python > Module Structure: Default project layout but basically, your module's pipelines need to be importable with import main. So if you kept the main package, just make sure you import your other file in __init__.py. If not, then you may need to adjust the config in pyproject.toml in order for the build backend to find your file. There's several examples of that in tests.

[deft delta]

I agree that telling people to put their code. in __init__.py flies in the face of Python convention. That file is supposed to be for housekeeping around imports, not business logic. https://docs.python.org/3/tutorial/modules.html

[alpine oxide]

Is anyone else at Pycon this weekend?

[junior girder]

Dagger doesn’t necessarily recommend putting your code in __init__.py, but I understand how you may think that way because of the default template.

There’s a need to balance pragmatism with the defaults here. Initially, the default Python template put the code in src/main.py. In the docs there was a section for breaking it into multiple files, and that was to convert the main python module into a package. Several people were confused and asked for help because they somehow missed the importance of importing their objects in __init__.py, or they just created multiple files under src, without the __init__.py and a parent main directory.

So I thought it would be best to start with a package which was done as part of adding uv support. As it’s an initial template, and to keep things simple, I didn’t see the need to break it further into two files to keep __init__.py just for imports. And compared to the other SDKs, Python already creates the deepest file structure:

• Go: main.go
• TypeScript: src/index.ts
• Python: src/main/__init__.py

Pragmatically, if there isn’t much code in it, I don’t see why not keep it in there. My thinking was that people would re-organize the starter package to their heart’s content. There’s a lot of flexibility here. Dagger only needs to import a module/package called main. It's installed with pip install -e . and it finds the functions via the import system. There isn’t an entrypoint and no other naming conventions.

What would make this better?

• Would it be enough to add a comment to the default template recommending to move the logic to another file and import the main object in __init__.py?
• Or would most Dagger pythonistas appreciate if the default template was already split in multiple files just to avoid having logic in __init__.py?
• Improve the documentation in Default project layout and Alternative project template ?

[deft delta]

I think improving the documentation and the template would suffice. Make the template import main.py or something from __init__.py

[junior girder]

Can you make a issue for it in GitHub?

[deft delta]

https://github.com/dagger/dagger/issues/7430

[steep widget]

I am still a bit confused about concurrent builds in python

I have this function

import dagger
from dagger import dag, function, object_type

@object_type
class MyModule:
    @function
    def build(self, src: dagger.Directory) -> dagger.Directory:
        """Build and return directory of go binaries"""
        # define build matrix
        gooses = ["linux", "darwin"]
        goarches = ["amd64", "arm64"]

        # create empty directory to put build artifacts
        outputs = dag.directory()

        golang = (
            dag.container()
            .from_("golang:latest")
            .with_directory("/src", src)
            .with_workdir("/src")
        )

        for goos in gooses:
            for goarch in goarches:
                # create directory for each OS and architecture
                path = f"build/{goos}/{goarch}/"

                # build artifact
                build = (
                    golang
                    .with_env_variable("GOOS", goos)
                    .with_env_variable("GOARCH", goarch)
                    .with_exec(["go", "build", "-o", path])
                )

                # add build to outputs
                outputs = outputs.with_directory(path, build.directory(path))
        
        return outputs

When I execute it it builds all 4 binaries at once

https://dagger.cloud/levs-test-org/traces/cb20872782cc368bc862ef71526659a0

I am not using task groups at all so its confusing why this works at all.

[junior girder]

This is the ideal way to use concurrency because it relies 100% on declaring the dag and letting the engine deal with the parallelism. Since you're creating files, it's totally correct to lazily add each to an empty directory and then return that directory. When it's needed (sync, export, entries...), it'll trigger the whole execution then.

If you're testing, that's different. You could build several containers to test different versions, for example, and store their output in an empty directory as well. Simply use the option in withExec to redirect output to a file, and return that file.

The problem is that depending on the case, this looks like a hack. Can definitely do it though, and in this build case, specifically, it's not a hack, it fits well with what you're trying to do. But otherwise, it's better if we have a better primitive to do it through the API. With interfaces, however, we'll be able to have a single core API function to sync multiple types of IDs, like any combination of Container, Directory or File for example.

In that case, you'd just do this:

dag.Sync(test310, test311, test312, lint)

Where those variables are lazy Container objects. That would run all of that in parallel, without needing to reach for the language's concurrency features.

[alpine oxide]

Just a heads up, I seem to be getting drawn in to help Jazzband reduce admin friction
https://github.com/jazzband/help/issues/366
and I may want to look into calling dagger pipelines from the flask app they currently use to release packages.
LMK if that sounds interesting to you. (If you use Django, it's probably interesting)

[junior girder]

Oh yeah, definitely interested! Longtime Django and Jazzband user here (not for the past 2 years though)!

[alpine oxide]

the other related discussion is https://github.com/jazzband/help/issues/364#issuecomment-2134771970 though that is more focused on security.

[alpine oxide]

I can't remember who was asking about automatically generating diagrams from python code, but here is a livestream of an effort to improve the clarity of the python core language reference. https://www.youtube.com/live/9NxAXjZBjbI?si=y_2fUt1DAac1tBZi

[stark dagger]

👋 @junior girder something to check tomorrow:

if I do a dagger install github.com/vito/daggerverse/testcontainers@a1647750b3ef09b83091b40185f498bb995ed540 in a python module, the generated setup function (https://github.com/vito/daggerverse/blob/a1647750b3ef09b83091b40185f498bb995ed540/testcontainers/main.go#L28) has the following signature:

    def setup(self, ctr: Self) -> Container:

which makes it not possible to be used in the with_ calls given that Self is not of type Container. Does it seem like a 🐛 or am I missing something here?

cc @deft delta

[stark dagger]

seems like Helder is out this week. Maybe @grave sinew has an idea on how this works?

[junior girder]

Looks like a bug. That argument shouldn’t be a Testcontainers (i.e., Self), should be Container.

[stark dagger]

awesome, thx for dropping the comment during your time off 🙏 . Will open an issue for this.

[steep widget]

Im running into a very strange issue, I have a module that is working just fine, but as soon as I import from dateutil import parser I get this error when I try to run it

Error: input: module.withSource.initialize resolve: failed to initialize module: failed to call module "nostalgia" to get functions: call constructor: process "/runtime" did not complete successfully: exit code: 1

Stderr:
╭─ Error ──────────────────────────────────────────────────────────────────────╮
│ The "main" module could not be found. Did you create a "src/main.py" file in │
│ the root of your project?   

[ionic hare]

Why does the Python client SDK expose eg. Container.from() as container.from_()? What's with the trailing _

[steep widget]

I am not 100% sure but its always been that way since I joined, even pre-zenith

https://archive.docs.dagger.io/0.9/sdk/python/628797/get-started#step-2-create-a-dagger-client-in-python

According to pep8 this is a python convention used to avoid conflicts with Python keywords

https://peps.python.org/pep-0008/#descriptive-naming-styles

[ionic hare]

ah the keywords! of course

https://tenor.com/view/iknewit-tommyboy-gif-25886935

[median plover]

how do I fix this on mac? I can't use docker desktop in my org. I have colima running and podman.

✘ connect 0.0s
! start engine: no fallback container found
  ✘ starting engine 0.0s
  ! no fallback container found
    ✘ create 0.0s
    ! no fallback container found
    ┃ 08:07:41 WRN failed to resolve image; falling back to leftover engine error="error getting credentials - err: exec: \"docker-creden
    ┃ esktop\": executable file not found in $PATH, out: ``"      

[junior girder]

Can you post this on https://discord.com/channels/707636530424053791/1030538312508776540 pls?

[stark dagger]

hey Helder, never got to open an issue about this. Not sure if this was under your radar. 🙏

[junior girder]

No, I was on PTO, so answered on my phone, abroad. Just came back and I'm bound to drop stuff from Discord. If you can make an issue, I'd appreciate it 🙏

[stark dagger]

Will do I'm a bit

[stark dagger]

https://github.com/dagger/dagger/issues/7652

[junior girder]

https://github.com/dagger/dagger/pull/7654 🙂

[high gust]

anyone experiencing pipeline kill cause of memory or cpu? version 0.11.7 getting
Process finished with exit code 137 (interrupted by signal 9:SIGKILL)
on a working pipeline

[ionic hare]

I'm working on a demo using the Python SDK, and error messages show me filenames that don't match my local filesystem, so I can't "click-open" from the terminal to the IDE. Has anyone else encountered this, and is there a fix?

eg.

│ /usr/local/lib/python3.11/importlib/__init__.py:126 in import_module         │
│                                                                              │
│   123 │   │   │   if character != '.':                                       │
│   124 │   │   │   │   break                                                  │
│   125 │   │   │   level += 1                                                 │
│ ❱ 126 │   return _bootstrap._gcd_import(name[level:], package, level)        │
│   127                                                                        │
│   128                                                                        │
│   129 _RELOADING = {}                                                        │
│ in _gcd_import:1204                                                          │
│ in _find_and_load:1176                                                       │
│ in _find_and_load_unlocked:1147                                              │
│ in _load_unlocked:690                                                        │
│ in exec_module:936                                                           │
│ in get_code:1074                                                             │
│ in source_to_code:1004                                                       │
│ in _call_with_frames_removed:241                                             │
╰──────────────────────────────────────────────────────────────────────────────╯
╭──────────────────────────────────────────────────────────────────────────────╮
│  /src/dev/src/main/__init__.py:27                                            │
│     def test(self, src dagger.Directory):                                    │
│                        ▲                                                     │
╰──────────────────────────────────────────────────────────────────────────────╯
SyntaxError: invalid syntax

[junior girder]

Most recent in the stack starts at the bottom. In the runtime container the context directory is mounted at ˋ/srcˋ. Seems like an issue with Go muscle memory because you're missing a ˋ:ˋ between the ˋsrcˋ parameter and its type: ˋsrc: dagger.Directoryˋ.

[ionic hare]

Thanks. I understand why it's happening, it's just that it breaks the "click from terminal to edit", was curious if there's a way to fix it, maybe by having python print a relative path in the trace?

[junior girder]

Ah, sorry about that. It's technically possible to rewrite those file paths in Python. However, I think I remember a feature in VSCode where you can map a directory in a container to the host. I believe it's to support the dev containers use case. Maybe it was PyCharm, I don't remember. But I think it would do what you're asking, like being able to click on a file path from the terminal in the IDE, and it would open the source file.

[junior girder]

Heads up, I'm updating the Python runtime module to support uv.lock. Not by default, but it'll use it instead of requirements.lock if it exists. Been using it on my python modules and it's great! Good job @tropic holly (and the Astral team) 🎉 Even uv run vim .. The default on .venv also helps with other IDEs.

I wonder how uv compares to rye when fetching toolchain. I'd love to use a wolfi base container and have uv download python, instead of using the larger python slim but last time I tried, it seemed to take twice the time. The advantage of fetching python on demand is it would make the runtime much simpler.

[junior girder]

Hmm, it's not that different, just ~1s slower with uv. This isn't very realistic, but it's enough to compare the time to download.

[junior girder]

The standalone that uv fetches is much larger.

• uv: 433MB
  • install: 234MB
  • build: 138MB
• python slim: 156MB
  • install: 30MB
  • build: N/A
    Related: https://github.com/astral-sh/uv/issues/4822

[tropic holly]

Interesting, thanks!

We shouldn’t be installing debug builds (that would be a bug) but IIRC we don’t strip them right now

[junior girder]

After https://github.com/astral-sh/uv/pull/4775, it's now:

• image size: 433MB → 110MB
• install dir: 234MB → 83MB
  Seems like a viable option now 🙌

[south geode]

@junior girder Any chance you have a poetry example of a module with multiple functions?

[junior girder]

I have this one: https://github.com/helderco/daggerverse/tree/main/poetry. Do you mean multiple files?

[south geode]

Well a bit of both, but I've just realised (I think) that a module can only have one class, but many functions within it?

[junior girder]

A module can have multiple classes, each with multiple functions. It's just that at minimum, one class must me named after the module, and its constructor becomes the "entrypoint" for the module.

You can see an example with multiple classes here: https://github.com/dagger/dagger/tree/main/sdk/python/dev/src/main

That's using uv instead of Poetry, but the diff is in pyproject.toml (and the lock file), not in code or file structure.

But you can see in these test cases what config you need in Poetry for different file organizations.

[short crow]

For anyone new to Dagger and wanting to use Python, this intro from @woven orchid is great. Thanks for creating it @woven orchid ! https://www.youtube.com/watch?v=jm61K6vWuqY&t=511s

[woven orchid]

Thanks for posting this!
My "slides": https://gist.github.com/KGB33/299a607966ab4eb8220bf6ca78e9ef41

[south geode]

Is the old school async with dagger.connection(): dag.container stuff gone for non-module usage? Getting errors with from dagger import dag unknown import symbol, and "connection is not a known attribute of module 'dagger'"

Oh, probably just python version errors, nevermind!

Still getting the diagnostics but the test runs.

[junior girder]

Yeah, in a module the connection is managed for you, so don't use dagger.connection(). Unless your function is building a container and that container is the one with the code that has dagger.connection(). The Python SDK's dev module (a Python Module), does that to run pytest, which does call dagger.connection() to test the generated client.

[south geode]

I'm not in a module here, this is real old school stuff from the pre-functions Dagger

Can you link that testing setup, sounds interesting

[junior girder]

Sure, this one tests for what you're looking for: https://github.com/dagger/dagger/blob/5d10da9749a93b39e4ea0d350fa49f3fb8c8ecd4/sdk/python/tests/client/test_default_client.py#L22-L29. It was implemented before modules (dag). Note that dag uses a shared connection and dagger.connection() is different from dagger.Connection. It's explained in https://dagger-io.readthedocs.io/en/sdk-python-v0.12.4/connection.html#dagger.connection.

Note that if using dagger.connection() in a single test case with the context manager, it'll close the connection in the end. But you can put it in a fixture, like this: https://github.com/dagger/dagger/blob/5d10da9749a93b39e4ea0d350fa49f3fb8c8ecd4/sdk/python/tests/client/test_integration.py#L16-L19. This way the same connection is shared for multiple tests which makes it more performant.

[onyx sandal]

Hi, anybody know if there is a solution to load a File as as dagger.File without using function call params, in order to pass this file to a container.with_file() method?

import dagger
from dagger import dag, function, object_type

@object_type
class Example:
    @function
    async def build(self) -> dagger.File:
        # For example like this:
        file = dagger.File("./pom.xml")
        container = (
            dag.container()
               .from("foobar")
               .with_file(file)
               .with_exec("/bin/sh", "-c", "build.sh")
        )

        return container.file(path="./build")

[ionic hare]

This will work, but it will load pom.xml from your function's runtime container (a sandbox).

• Sometimes that's what you want, for example if you just called a native library that wrote that file to the local filesystem.
• But if you're looking to access pom.xml on the client's local filesystem (outside the sandbox) then that's not supported.. yet! Here's the issue tracking that: https://github.com/dagger/dagger/issues/7647

[onyx sandal]

Hi @ionic hare ok so I will subscribe to this Issue and work with dagger call params until a solution is found.Thanks!

[mental silo]

Hey folks, I am newbie in Dagger and would like to do my first PoC with below use case via Github Actions..
How can I create one reusable module with Dagger to cache entire venv directory pip install ?

  with:
    python-version: '3.10'
    cache: 'pip'
    cache-dependency-path: |
      scripts/dev_requirements.txt
      scripts/requirements.txt
      scripts/backend_pmf_test/requirements.txt
      scripts/check_pagerduty/requirements.txt
      scripts/check_vault/requirements.txt
      scripts/jira/requirements.txt
      scripts/nkw_versions/requirements.txt
      scripts/quality_comparison/requirements.txt
      scripts/utils/docgenerator/requirements.txt
      scripts/utils/emulate_traffic/requirements.txt
- name: Run pip install
  run: |
    pip install \
        -r scripts/dev_requirements.txt \
        -r scripts/requirements.txt \
        -r scripts/backend_pmf_test/requirements.txt \
        -r scripts/check_pagerduty/requirements.txt \
        -r scripts/check_vault/requirements.txt \
        -r scripts/jira/requirements.txt \
        -r scripts/nkw_versions/requirements.txt \
        -r scripts/quality_comparison/requirements.txt \
        -r scripts/utils/docgenerator/requirements.txt \
        -r scripts/utils/emulate_traffic/requirements.txt

[steep widget]

This is a good use case for cache volumes, you can learn more about that here in the docs: https://docs.dagger.io/manuals/developer/cache-volumes/

FYI, you can also use the -r flag inside of one of your requirements.txt file to reference other files, this way you can drastically simplify your install command

[mental silo]

thanks @steep widget ,but how should I call this from github action workflow? Can I parametrized the volume path where i can override when I call module or should I add hardcoded?

[steep widget]

Ah sorry, you may run into issues here because these cache volumes cannot just be saved as files in GitHub Actions.

Distributed caching on ephemeral runners is still a work in progress, the current solution is to either use the experimental cache on dagger cloud, or consider running a long lived dagger engine yourself that your runner points to.

[mental silo]

@steep widget is there any ETA for this? What I understood, if Dagger can provide this, we can replace https://github.com/actions/setup-python and even can extend based on our needs, am I right?

[steep widget]

No ETA but yeah I think it will make running dagger on vanilla CI runners at GHA, CircleCI, GitLab and likely a few others much smoother in the future

There is a high level issue tracking this sort of work here: https://github.com/dagger/dagger/issues/8004

[mental silo]

thanks @steep widget , I will have a look.
So then currently, cacheVolume is store the file in Dagger cloud, right? But what about local? When I ran dagger in my local, is it also pulling from Dagger?
Or use localfilesystem with Buildkit

  ): Container {
    return dag
      .container()
      .from("node:21")
      .withDirectory("/src", source)
      .withWorkdir("/src")
      .withMountedCache(
        "/src/node_modules",
        dag.cacheVolume("node-21-myapp-myenv"),
      )
      .withMountedCache("/root/.npm", dag.cacheVolume("node-21")). ->>> here
      .withExec(["npm", "install"])
  }
}

[steep widget]

thanks @Lev Lazinskiy , I will have a

[fringe relic]

I am not sure if this has been answered before, if it has, i am not able to find it.

Right now, I am writing dagger functions but I am not sure how to write tests for them. I am doing a POC where these functions will be used by multiple teams in their CI pipelines and I want to ensure changing them will not break those pipelines.
I am writing these functions in python and would be interested in a pythonic way of testing them.

[ionic hare]

This is an active topic of discussion, see #1275901633880129749

[ionic hare]

What's the difference between @dagger.function and @class_method? I see both in the Python SDK source code

[junior girder]

@classmethod is native Python for methods that belong to the class instead of the instance. They are more commonly used as factory methods to create instances of a class and should not be exposed to dagger as functions. The only exception is a class method named create by convention, to use as the module’s entrypoint (main object constructor), when the factory pattern is necessary, or async.

[silver warren]

Hi, I'm trying to do a function in python (I did some functions in golang) but I have some issues.
The first one is I'm using the __init__ in order to do like the new function in go.
In the __init__ I'm create a base container.
I have something like that:

@object_type
class Python:
    def __init__(
            self,
            pipeline_id: Annotated[str, Doc("Kind of namespace between different projects")],
            source: Annotated[dagger.Directory, Doc("Source directory")],
            exclude_directories: Annotated[list[str], Doc("Directory exclusion pattern")] | None,
            exclude_files: Annotated[list[str], Doc("File exclusion pattern")] | None,
            build_system_packages: Annotated[list[str], Doc("All required system packages for build time")] | None,
            runtime_system_packages: Annotated[list[str], Doc("All required system packages required after build time")] | None,
            python_version: Annotated[str, Doc("The python version to use")] = "3.12.5",
            poetry_version: Annotated[str, Doc("The poetry version to use")] = "1.8.3",
    ):

So with that when I doing a dagger call --help I can see these params in the output but when I'm trying to call a function called test with an optional arg I have the following error:

TypeError: Python.__init__() missing 6 required positional arguments: 
'pipeline_id', 'source', 'exclude_directories', 'exclude_files', 
'build_system_packages', and 'runtime_system_packages'
Run 'dagger call test --help' for usage.

Someone have an idea what I'm doing wrong ?

Also I tried to do like that:

@object_type
class Python:
    pipeline_id: Annotated[str, Doc("Kind of namespace between different projects")]
    source: Annotated[dagger.Directory, Doc("Source directory")]
    exclude_directories: Annotated[list[str], Doc("Directory exclusion pattern")] | None
    exclude_files: Annotated[list[str], Doc("File exclusion pattern")] | None
    build_system_packages: Annotated[list[str], Doc("All required system packages for build time")] | None
    runtime_system_packages: Annotated[list[str], Doc("All required system packages required after build time")] | None
    python_version: Annotated[str, Doc("The python version to use")] = "3.12.5",
    poetry_version: Annotated[str, Doc("The poetry version to use")] = "1.8.3",

    def __init__(self):

But my args are not recognized from cli

[junior girder]

Python __init__()

[fringe relic]

Instead of | use = None

That will solve the issue.

[late hornet]

How do i remove the arguments from the docs which i dont want people playing with (they were kinda private)

https://daggerverse.dev/mod/github.com/pjmagee/dagger-badge@262fbd7bcd8c80b4862b05983ec5faf64fdcf6b3

The only one which someone should be using is --raw-url string add-to-readme --file file:path

python isnt my naturla lang, do i add __ to the variables, will dagger ignore those?

dagger call -m https://github.com/pjmagee/dagger-badge --raw-url=https://example.host/raw-path-to-your/dagger.json link will give you a md link that looks like this:

[late hornet]

thought i'd fix it with dataclasses.initVar but those got exposed still.

nevermind, i fixed it with my first thought... which i didnt try earlier because i obviously wanted to make it difficult for no reason... __ has fixed it for me

[junior girder]

For fields , as long as you don’t use dagger.field() they won’t be exposed as a function. However, they’ll be used as constructor arguments by default, so if you also don’t want an field as a constructor argument, you can flag it with dataclasses.field(init=False).

For example:

@object_type
class DaggerBadge:
    segment_prefix: str = dataclasses.field(init=false, default="![Dagger]")

That way segment_prefix won’t show up as a constructor argument nor as a getter function, but should still be initialized with that default, making it internal to Python only.

[junior girder]

dataclasses.InitVar is for making constructor-only arguments, i.e., won’t be attributes in the object after init.

[scenic escarp]

Hello there,
is it possible to set up dagger, that it does not activate initialize phase after a code change?

# this is the first call
✔ connect 0.7s
✔ initialize 19.8s

# this is the second call (third, etc...), no change in python code was made, it is fast
✔ connect 0.7s
✔ initialize 2.3s

this is the call, when I changed commentary in python code, dagger is again initializing python...
✔ connect 0.7s
✔ initialize 20.9s

I am using plain python, there are not any additional python modules included. What am I doing wrong? It is bit annoying. Thank you

[junior girder]

The initialize phase in the CLI refers to loading the module. By default it uploads (files from host to engine) everything inside it, including .venv and the generated sdk. You can add these to dagger.json under "exclude": [".venv", "sdk"] but see if you have more stuff that could be re-uploading.

[junior girder]

Something else that might help understand the diff between one change vs the other is that a change in function signatures and descriptions, or basically anything that changes what's reported to dagger, will generate a different ModuleID, which busts the cache on more stuff than just a change in a function body. This means that codegen needs to rerun for example.

Still, 20s tells me you're probably uploading a bunch more stuff than needed.

This should make it better: https://github.com/dagger/dagger/issues/6627. There's also some caching issues with the Python SDK that I need to continue to hunt down, but should get into that pretty soon.

[scenic escarp]

       ✔ Container.withExec(args: ["uv", "pip", "compile", "-q", "-o", "requirements.lock", "sdk/pyproject.toml", "pyproject.toml"]): Container! 3.2s
          ✔ exec uv pip compile -q -o requirements.lock sdk/pyproject.toml pyproject.toml 3.1s
        ✔ Container.directory(path: "/src/6fc4e5716b471a24a064bbfe771ad4a9cf08b73cace0c15b281a69a1a25b2f8e"): Directory! 3.2s
        ✔ exec /runtime 0.7s
        ✔ Container.withExec(args: ["uv", "pip", "install", "-e", "./sdk", "-e", ".", "--no-deps", "-r", "requirements.lock", "--compile-bytecode"]): Container! 5.9s
          ✔ exec uv pip install -e ./sdk -e . --no-deps -r requirements.lock --compile-bytecode 5.9s
        ✔ Container.withExec(args: ["uv", "pip", "check"]): Container! 8.2s
          ✔ exec uv pip check 2.3s
        ✔ Container.withEntrypoint(args: ["/runtime"]): Container! 8.2s

the most time is consumed by uv downloading python dependencies

[junior girder]

Heads up everyone for https://github.com/dagger/dagger/pull/8311. dagger init --sdk=python will generate a uv.lock file in next release.

[scenic escarp]

@dense surge Hi look here in this channel for more details, I have not tried tracing so far, I will try it tomorrow. I will send tracing info there.

[ionic hare]

Is it normal to have the main logic of a Python module in __init__.py? The last time I was actively developing in Python was 10 years ago, but back then it was definitely not normal

Can someone explain to me the difference between:

dagger call -m github.com/dagger/dagger/sdk/python/runtime codegen

and:

dagger call -m github.com/dagger/dagger/sdk/python/dev generate ?

What is the purpose of each, and how do they interact?

EDIT: Ah, I guess -m dev generate is for generating the official core API bindings. Whereas -m runtime codegen is for generating bindings for each module.

But, shouldn't they share a common implementation?

[steep widget]

No, this is not normal, even in our code gen we have this snippet, I think instead of this snippet we should put the code in a main.py file (or something) and import it from init.py

# NOTE: it's recommended to move your code into other files in this package
# and keep __init__.py for imports only, according to Python's convention.
# The only requirement is that Dagger needs to be able to import a package
# called "main", so as long as the files are imported here, they should be
# available to Dagger.

[junior girder]

I've planned to do this for the longest time, but after allowing for a Python package other than "main" (see this comment). That's create the following file structure:

.
└── src
   └── my_module
      ├── __init__.py  # for imports
      └── main.py      # for main object

[junior girder]

Is it normal to have the main logic of a Python module in __init__.py?

I responded to that in issue #7439. To reiterate, you may remember that in the beginning the template generated a src/main.py file, and the documentation had instructions on how to break a big file into multiple ones. It's just like any other Python project where you need to have a __init__.py to make it a package.

The problem is that I noticed multiple people asking for support anyway in how to use multiple files as they weren't able to do it on their own, so I just bit the bullet and made it the default. Now the problem is that for those that don't need multiple files, they still leave all the code in __init__.py, even with the comment in the template recommending to put their functions in other files.

So there's changes coming that pulls several things together here:

• Use the Dagger Module's name as the Python module's name, instead of always being "main"
• Generate the template in a src/<module_name>/main.py, and import it in src/<module_name>/__init__.py
• Generate a src/<module_name>/__main__.py for the Module's entrypoint instead of the implicit import main from the dagger-io library, or add an entrypoint entry in pyproject.toml

These changes will bring a lot of flexibility to users on how their Module is initialized, while providing better defaults.

[junior girder]

As for the runtime codegen vs dev generate, you're right except that the runtime does more:

• The dev Module's generate Function is used for generating the client bindings for the core API, to be published in PyPI (used to be in /ci/python_sdk.go, but was reimplemented here)
• The runtime Module's codegen Function implements half of the required interface for SDKs today, but it's for generating the files for supporting Dagger Modules:
  1. Gets the files in the source directory
  2. Adds the SDK's client library
  3. Re-generates the client bindings for the client library, based on current API schema
  4. Adds any files from the template if it's a new module
  5. Updates the lock file (without upgrading packages)

Only 3) calls to the same code that dev's generate does.

[ionic hare]

would it make sense to split the template creation into its own function? would move logic into the engine, and make it easier to iterate

[junior girder]

Yeah, there's several things like that that I want to improve on.

[scenic escarp]

hello,
is it possible to change source of ghcr.io/astral-sh/uv image?
https://github.com/dagger/dagger/blob/f1746e09a0e13caa53029805148f818b7f20766c/sdk/python/dev/src/main/__init__.py#L11
there is os.getenv("DAGGER_UV_IMAGE", "ghcr.io/astral-sh/uv:latest") function but setting env DAGGER_UV_IMAGE=ghcr.io/astral-sh/uv dagger call does not do anything. Maybe custom dagger engine image container with env variable setup inside of it could do the trick? 🤔

[junior girder]

Hmm, is that because you need a private company mirror or similar?

[scenic escarp]

yes this is the reason, another complication is that there is .../mirror/... added to the URL docker.company.com/mirror/astral-sh/uv

[scenic escarp]

I am not sure if the trick with https://docs.dagger.io/manuals/administrator/custom-registry/ has some option to add this mirror thing into the URL

[junior girder]

I was just going to ask if that doesn't work for you. I don't think it matters if you have an extra "mirror" in the URL.

[scenic escarp]

[registry."ghcr.io"]
  mirrors = ["mirror.foo.com/mirror"]

or maybe ?

I did not try it so far, I was just exploring options

[junior girder]

Oh, maybe it's just for the domain? I'm not sure, need to check.

[junior girder]

Yeah, maybe that works. Do you know @stark dagger?

[stark dagger]

yes, this should work

let us know @scenic escarp if you're having issues with that particularly as the buildkit logs regarding mirrors are quite scarse 🙏

[scenic escarp]

unfortunately it does not work 😦

$ docker exec 7431e017b6a5 cat /etc/dagger/engine.toml
debug = true
trace = false
insecure-entitlements = ["security.insecure"]

[registry."docker.io"]
  mirrors = ["docker.company.com/upstream"]

[registry."ghcr.io"]
  mirrors = ["docker.company.com/upstream"]

        ✔ Container@xxh3:5424b78ee25d59fa.withEnvVariable(name: "DAGGER_UV_IMAGE", value: "ghcr.io/astral-sh/uv:0.4.9@sha256:d8a3dbcb14b1e20a5e8baafe8dc9981e9bec88a3eef6b24f5392770015502071"): Container! = xxh3:18f981772206fe0b 0.2s

[scenic escarp]

injecting env variable into the custom-engine did not help either 😦

exec 3460c59893fc env
docker exec 3460c59893fc env  | grep DAGGER_UV_IMAGE
DAGGER_UV_IMAGE=docker.ops.iszn.cz/upstream/astral-sh/uv:latest

[junior girder]

That env var is informational, it doesn't do anything.

[junior girder]

But it happens after pulling uv, which tells me you didn't have an issue, right?

[junior girder]

You should see something like this:

    ✔ Container.from(address: "ghcr.io/astral-sh/uv:0.4.9@sha256:d8a3dbcb14b1e20a5e8baafe8dc9981e9bec88a3eef6b24f5392770015502071"): Container! 0.0s
      ✔ resolving ghcr.io/astral-sh/uv:0.4.9@sha256:d8a3dbcb14b1e20a5e8baafe8dc9981e9bec88a3eef6b24f5392770015502071 0.0s
      ✔ cache request: pull ghcr.io/astral-sh/uv:0.4.9@sha256:d8a3dbcb14b1e20a5e8baafe8dc9981e9bec88a3eef6b24f5392770015502071 0.0s

Which should appear close to the base image for the container itself:

    ✔ Container.from(address: "docker.io/library/python:3.12-slim@sha256:8ac54da5710cdd31639bb66f5bc1888948fc2866c0b5b52913b4b33d8252e510"): Container! 0.0s
      ✔ resolving docker.io/library/python:3.12-slim@sha256:8ac54da5710cdd31639bb66f5bc1888948fc2866c0b5b52913b4b33d8252e510 0.0s
      ✔ cache request: pull docker.io/library/python:3.12-slim@sha256:8ac54da5710cdd31639bb66f5bc1888948fc2866c0b5b52913b4b33d8252e510 0.0s

The code that makes that happen is here: https://github.com/dagger/dagger/blob/e31b9e9b0312a95bde2635adeb3cbfe0ce6f5a0f/sdk/python/runtime/main.go#L218

As you can see, the DAGGER_UV_IMAGE is added afterwards: https://github.com/dagger/dagger/blob/e31b9e9b0312a95bde2635adeb3cbfe0ce6f5a0f/sdk/python/runtime/main.go#L224

That allows me to use the same image for pulling uv (including digest) in a Python module:

• https://github.com/dagger/dagger/blob/e31b9e9b0312a95bde2635adeb3cbfe0ce6f5a0f/sdk/python/dev/src/main/__init__.py#L11
• https://github.com/dagger/dagger/blob/e31b9e9b0312a95bde2635adeb3cbfe0ce6f5a0f/sdk/python/dev/src/main/__init__.py#L89

[ionic hare]

Very weird error while developing on our CI:

dagger install ../sdk/python/dev fails with:

warning: `uv run` is experimental and may change without warning
thread 'main' panicked at crates/uv-resolver/src/lock.rs:402:18:
could not find root
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

But I have not modified ../sdk/python/dev...

If I install the main version it works:

dagger install github.com/dagger/dagger/sdk/python/dev -> no error

Ah, this fixed it: git fetch origin main:main; git checkout -f main -- ../sdk/python/dev

[hybrid cape]

Just FYI I went down a similar path and was able to get this to work. I had to use crane to get the exact sha into the mirrors

[scenic escarp]

Hello Chad, I do not know crane (a docker compose alternative?) How can it help? Can you provide some quick example please?

[scenic escarp]

Technically while everything works, it is our company good practice to store external packages, images etc in the our infrastructure

docker images are a bit special case, the are not automatically routed through our internal mirror/repository I have to manually create replication, ask security to accept it and then the infrastructure team finally downloads and stores image in the our internal repository/mirror

thats why am I dealing with additional .../upstream/... in the docker image URL. It is pointing out that image comes from the "internet"

[junior girder]

Private mirror for ghcr.io

[hybrid cape]

crane copy --insecure docker.io/library/python:3.11-slim@sha256:3f3c35617e79276c5f6a2e6a13cdbabdd10257332df963c90c986858b26fad5e localhost:5000/library/python:3.11-slim@sha256:3f3c
35617e79276c5f6a2e6a13cdbabdd10257332df963c90c986858b26fad5e

It isn't a docker compose alternative. It is a tool for managing docker registries. Copying data between etc.

[south geode]

Getting Skipped 1 function(s) with unsupported types: when using dagger functions after updating to 0.12.7 and updating a dependency, the only arg types are dagger.Secret and str, what am I not seeing?

[south geode]

Skipped 1 function(s) with unsupported types

[still crag]

Hi there! I'm trying to reproduce the cache volume re-use trick of @stark dagger k3s module . This is my current best attempt https://gist.github.com/dciangot/3a2878753316113c0d51b01bd86eef1b , calling create-vm and then get-vm-id with the same --name option, I find /cache in the latter even though /opt/cache/vm_id.json is correctly created in the previous call. Any idea of what I'm doing wrong? Is this even possible to do in Python SDK?

[stark dagger]

should work. Checking really quick

[stark dagger]

ok, found the issue. It works if the owner of the with_mounted_cache is the same.

now trying to understand if that was an intentional restriction or a buildkit thing

[still crag]

awesome, thx!

[stark dagger]

silently pinging @north cypress and @willow sail as I can't remember the buildkit limitations here. This sentence applies to what's happening here though: "It does not have any effect if/when the cache has already been created." I guess the confusing part is what "has already been created" actually implies.

[still crag]

not that it was needed, but I confirm that now everything is working

[willow sail]

so i think what's happening here is that the owner is applied to the source - which is the "base" state of the cache directory

but once the cache exists, the owner isn't going to be set

i'm very confused honestly by the interactions

i've never touched this logic

[still crag]

anyone with some ref on how to put examples in a module written in python SDK?

[junior girder]

Daggerverse examples

[modern dove]

Hi everyone. I've got a question about the "dagger" way of doing something.

I have a python fastapi web app. I've got a package function that returns a dagger.Container (using the Python SDK).

The issue is that the as part of the webapp startup, I connect to a database (which is an external DB like postgres/a managed servce on the cloud provider). So dagger correctly builds the docker image, and then proceeds to start it up using the exec command (basically uvicorn). Uvicorn starts but then the app will fail because it cannot connect to the database.

My question is: How would one use dagger to build such an image? Should I have not declared with_exec() to start uvicorn? That's what I would do with the Dockerfile.

Thanks in advance!

[modern dove]

ok, I think I've answered my own question here but I finish off the command with .as_service()?

[south geode]

Uvicorn starts but then the app will fail because it cannot connect to the database.

Your app connecting with a database doesn't really have anything to do with Dagger, configure your app to connect however you see fit (typically env vars for databases?)

If you want the app itself to remain alive, then yes, turn it into a service and expose it to other containers (with_service_binding) or to the host (app exposes one or more ports with with_exposed_port and returns dagger.Service, run call app up)

[modern dove]

Thanks very much MB

[south geode]

You may well have seen it already, but https://docs.dagger.io/manuals/developer/services has some python examples

[modern dove]

one more question, I now get this error:

Error: invalid selection for command "package": response from query: input: container.from.withExec.withWorkdir.withDirectory.withEnvVariable.withEnvVariable.withEnvVariable.withEnvVariable.withExec.withExec.withExposedPort.asService.sync resolve: IDFor: Service has no such field: "sync"

[modern dove]

No I didn't, but thanks - will take a look.

[south geode]

Does your service container method chain end with with_exposed_port(<something.).as_service()?

[modern dove]

yes, those are the last 2 lines I use

[south geode]

And you're not using .sync() anywhere? Check the order of commands as well - I've had mine in the wrong order before and had similar issues. As many with_exec() as you need, then end with with_exposed_port().as_service()

The function also needs to return dagger.Service not dagger.Container

[modern dove]

Oh apologies, just noticed I did not declare the method with async. Rookie mistake!

Thank you again

[steep widget]

@modern dove also be sur eto read this when you get a chance

https://docs.dagger.io/manuals/developer/services/#expose-host-services-to-functions

Dagger Services can be Container to Container, Container to Host, and Host to Container

This allows you to model out your reality in an elegant way using these first-class primitives

[steep widget]

I am going to find a way to turn this into a proper doc, but in the meantime, since we run into this a lot I wanted to share my current understanding of how to write a well structured python module that does not keep all the code inside of __init__.py

--

Here's the simplest approach

Given a module called MyModule

1. rename __init__.py to something (lets say main.py)
2. Create new __init__.py that only says

from .main import MyModule

I would put all the top level functions into main.py and then if you want to use other utilities or stuff you can then have as many files that you want that you import the same way inside of your main.py

So if I had a mysql_service function in db.py I could import with from .db import mysql_service inside of main.py

You can also create subfolders

levlaz@Levs-MacBook-Pro src % tree 
.
├── db
│   ├── __init__.py
│   └── service.py 
└── main
    ├── __init__.py
    └── main.py

Given this layout I can do somethign like this

from db.service import mysql_service

The empty __init__.py is the secret sauce that turns any directory into a python module

[next mortar]

Dagger module inside a regular app-repository - IDE completions / different virtual environments

I'm working on a regular python app project (pyproject, venv managed by uv) and want to introduce Dagger in that same repository so that I can run different build steps specific to this project. Assume it's not enough to just dagger call an existing module from daggerverse via cli. I want to provide some project specific wrappers and want to have these CI "scripts" version controlled inside the same repository.

If I dagger init into ./dagger, this creates an isolated python project with the src/main/__init__.py and the sdk/ folder with the generated code - all inside the ./dagger directory. Executing the module via Dagger CLI works as expected but it lacks proper IDE navigation / auto-completion in PyCharm because I'm running in the context of the regular app project and its outer venv.

What's your preferred way of dealing with this situation? I think it's correct that the dependencies used for the dagger module do not pollute the outer venv of the app. Still, having proper auto-completions for the Dagger part is also essential. I guess this would work by importing the ./dagger folder as a separate project in PyCharm. Still I think it makes sense to have just one instance with both app and ci code.

Thanks for some input. (Python is not my main language but I already spent quite some time reading about proper project structuring thanks to the folks from astral/uv)

[junior girder]

PyCharm monorepo

[steep widget]

Do we have any plans to support interfaces in python? I think that would be the proper solution for this:

I would love to be able to create a high level module that exposes other (cross langauge) installed modules through a unified CLI

Something like this:

from dagger import dag, function, object_type, PythonChild, GoChild, TypeChild

@object_type
class Parent:
    @function
    def py(self) -> PythonChild:
        return PythonChild()
    
    @function
    def go(self) -> GoChild:
        return GoChild()
    
    @function
    def ts(self) -> TypeChild:
        return TypeChild()

Where PythonChild, GoChild, TypeChild are three different installed modules in threee languages.

I would then be able to do something like dagger call py foo where foo is a function of PythonChild

I believe this is already possible in go with interfaces, is there any other approach that is possible with Python today?

[junior girder]

Yeah, it’s been almost done for a while. Just need a bit of time to finish. Including finding out what’s missing: 😄 https://github.com/dagger/dagger/pull/8097

[steep widget]

This is great!

[steep widget]

Python 3.13 came out - I love the better error messages feature, would love to see something like this applied to Dagger as well in the future (not just for python of course)

https://docs.python.org/3.13/whatsnew/3.13.html#improved-error-messages

[pulsar vault]

is the uv cache dir a good candidate to be mounted as a dagger cache volume ? or does it need more testing

[junior girder]

Yes, of course. The Python runtime uses it since day one of uv support.

[crisp marten]

@junior girder I can help test.

[junior girder]

@crisp marten do you feel up for https://github.com/dagger/dagger/issues/8533 ?

I'm annoyed since the beginning of the Python SDK that I haven't been able to make the type checker a part of linting (because there's always been errors). Fixing it requires a deeper knowledge of the SDK though.

[ionic hare]

I'm curious how you all feel about this? Should the Python SDK prioritize the best DX, or flexibility? https://github.com/dagger/dagger/issues/8689

[crisp marten]

yes, I just have a few questions. can wee meet briefly

[junior girder]

yes, I just have a few questions. can

[still crag]

hi there, any idea why I'm failing to make this appear as example in daggerverse? https://daggerverse.dev/mod/github.com/dciangot/daggerverse/shadeform@77a10bd443114753ae582ee99e6c583043ccad36 ? What am I doing wrong?

I tried different things, but this is the closest I get to this example https://github.com/marcosnils/daggerverse/blob/main/gptools/examples/python/src/main/__init__.py

[junior girder]

Ping @stark dagger

[stark dagger]

hey @still crag ! have you tried naming the example function shadeform__create_n_check?

the double underscore in the python case is intended to be used to differentiate between the module and the function name

[still crag]

I tried different combinations of double _ let me check

[still crag]

Unbelievable... I did have a commit with the correct syntax, but I never pushed it... it's working, sorry for the noise

[stark dagger]

np 🙌

[silver warren]

Hi, I have a strange behaviour in a custom python function, I have the following error:

Error: input: module.withSource.initialize resolve: failed to initialize module: failed to call module "python" to get functions: call constructor: process "/runtime" did not complete successfully: exit code: 1

Stderr:
Traceback (most recent call last):
  File "/runtime", line 5, in <module>
    from dagger.mod.cli import app
ModuleNotFoundError: No module named 'dagger'

Before some changes where I was trying to add a third party package it was working, then I tried to stash in order to come back to my previous step and remove the sdk folder but I still have the issue.
Someone has an idea what can I do with this error ?

[silver warren]

Hi, I have a strange behaviour in a

[silver warren]

Hi, I'm tyring to implement something similar to this example: https://docs.dagger.io/cookbook#terminate-gracefully but when I'm doing that I have this error: Function execution error: object PythonTestResult can't be used in 'await' expression any idea what is wrong ?

[junior girder]

Terminate gracefully

[silver warren]

Hi, not sure if I should post it to this channel but I have a strange effect with a golang function and when this function is called from python code.
I added a new call in a function when I'm doing my test from my golang code I can validate it's working as expected but when I'm trying to use it from my python code, it's saying me the object has no attribute

╭─ Error ──────────────────────────────────────────────────────────────────────╮
│ Function execution error: 'Gha' object has no attribute                  │
│ 'with_existing_pipeline'                                                 │
╰──────────────────────────

[glacial lance]

Hi! Is there a way to set a breakpoint (import pdb; pdb.set_trace()) when developing a local dagger module with python sdk and dagger call?

[junior girder]

Hi! Is there a way to set a breakpoint (

[junior girder]

Hi! 👋 I’m looking for an opinion from Pythonistas on default project structure of a Python Dagger module. This is related to the allow module package names other than main - #8709 PR.

The current approach when creating a module is to create a Python package (i.e., src/<pkg>/__init__.py) by default because of two reasons:

1. It’s detected automatically by most build backends (i.e., no extra config in pyproject.toml)
2. I found that users trip up when trying to split a module into multiple files and thought it makes it easier to either scale or trim

However, due to a few conversations lately, I’m wondering if most <@&1113692108755308593> users would rather have a single src/main.py or main.py module again by default, instead of the src/<pkg>/__init__.py package.

EDIT: To be clear, this is just about the default template for new modules, none of this is enforced after that and you're free do use any file structure that suits your needs.

1. Default today

my-module
├── src
│  └── main
│     └── __init__.py  # module doc and example functions
└── pyproject.toml     # (name = "main")

2. New default in upcoming v0.14 (PR)

my-module
├── src
│  └── my_module
│     ├── __init__.py  # just module doc and import
│     └── main.py      # example functions
└── pyproject.toml     # (name = "my-module")

3. Simplify?

Simpler file structure, but not following best practice:

my-module
├── main.py
└── pyproject.toml

Requires extra config in pyproject.toml:

[tool.hatch.build.targets.wheel]
packages = ["main.py"]

[silver warren]

I think it's better to follow common practices from python habits. For an example integration with IDE will be easier.
ok the structure is a bit complex but no so much imo

[forest nebula]

[7YOE, 5 professionally] I prefer multiple files over single files when working with python but I don't think we should enforce a rule.

I think init has its place and is here for a reason. it's great to expose "high level exported classes and structures"

[junior girder]

To be clear, none of this is enforced. I'm just talking about the default template for a new module, but you can modify that to suit your needs, including adopting any of these patterns.

[south geode]

The default coming in 0.14 looks good to me.

I presume Dagger will soon be able to use any module name, not just "main* - how will we configure that?

[junior girder]

By default, the Python SDK will look for a package named after the name in pyproject.toml. So if you have this:

[project]
name = "friendly-bard"

Then the SDK will look for the following:

• friendly_bard:FriendlyBard
• main:FriendlyBard (fallback)

So if you just make sure to import the main object (FriendlyBard) in src/friendly_bard/__init__.py, nothing else needs to be done.

If this doesn't work for you, you can explicitly specify where the main object is, with this entry point:

[project.entry-points."dagger.mod"]
main_object = "<module path>:FriendlyBard"

[south geode]

Sounds good to me. Those options will need to be easy to spot in the docs, easy to forget

[lucid bobcat]

Why not both?

For an easy way to get started using a single, simple import, then __init__.py is perfect. Once you start needing multiple imports then importing explicitly from specific modules is fine.

Definitely don’t like the main package default that currently exists.

[junior girder]

There can be only one default. As I said, this is only about the initial template for a module. You can change it after that.

[south geode]

Quick one that probably doesn't need a whole help topic: capturing stdout (which in my case is a string output from a command( and assigning to a variable results in... A tuple? Empty second element, but this has stalled me for half an hour

[south geode]

Unrelated follow-up, any examples of context directory in use in Python?

Found it: https://docs.dagger.io/api/arguments#directories-and-files 🙂

[river crane]

I can't remember whether this was mentioned in the documentation or a GitHub issue, but I recall someone stating that modules should be kept small and concise. To me, having a single src/main.py fits this philosophy best. With version 0.14 supporting module names other than main, it would make it even easier to spot the right file in the explorer (e.g. .dagger/src/my_dagger_module.py). Today, I often find myself refactoring the default structure to have that single src/main.py again, and only occasionally use the default structure when I need to build a larger module.

[junior girder]

Yes, but the reason for wanting to keep Dagger modules small is to avoid overwhelm on (especially) new users. That's why I'm wondering what's the better choice for a default. On one hand, if a Python module looks like a normal Python project, it'll look familiar to Python users and tooling. On the other, "simplifying" the file structure (placing main.py closer to pyproject.toml, and removing the __init__.py) requires extra config or Dagger specific handling for the SDK library to find the code, because it deviates from the standard.

We could default to a script (PEP 723), which is both simpler and also a (new) standard. Would have to rely more on docs to split into multiple files. Wonder if that's more work in most cases vs trimming down from a normal package (with __init__.py).

Anyway, it's a bit late now to change course as we're planning to release this week. I'll push the new file structure for v0.14, and create a GH issue afterwards with this question to present options and gather feedback. If most people want a simpler main.py, without the __init__.py package, maybe in v0.15 we change the default again.

[swift iron]

does anyone know if there's an easy way to create a dag.File from a string within python then mount it into my container?

[junior girder]

does anyone know if there's an easy way

[cinder jetty]

Option 2 looks good to me

[glacial lance]

I'm having a problem with ghcr.io/astral-sh/uv container used by python sdk. I'm trying to use dagger in a very locked down environment and one can not simply use any outside registry. Now I tried but setting to engine.toml and set our mirror, but there is a problem since our mirror requires login credentials.

As far as I know there is no way to provide login credentials via mirrors in engine.toml , but I might be wrong. Am I wrong and there is a way?

[junior girder]

Mirrors

[elder helm]

Dagger is not updating .npmrc file

[short crow]

PyCon Submissions are due Dec. 19th! https://pretalx.com/pyconus2025/

If you submit anything related to Dagger, please let us know. We'd love to support and showcase your CFP!

[hasty raven]

Hello

I'm trying to implement official dagger-cli analogue in python as we need a lot of bootstrap before Dagger call: setup tunnel to our k8s, login to Azure and container registry and some other things.

And I'm stuck with implementing dagger -m "[e-mail]:{MY_ORG}/{MODULE_REPO}.git" call {MODULE_FUNC}

What I'm doing:

async with dagger.connection(cfg):
    m = dag.module_source(f'[e-mail]:{my_org}/{module_repo}.git').as_module()
    m = m.initialize()
    await m.serve()

But I don't see my module in GraphQL
gql_schema = await dag._ctx.conn.session.get_schema() doesn't show my module functions (I can see them in result of await dag.current_type_defs())

Can somebody hint me what I'm doing wrong? How does dagger-cli register external modules in engine?

[hot anvil]

hello!
is there any way to use a private python package from github?
we have a private repo what i want to use in dagger, but i did not found a solution for that 😅

[junior girder]

Private Python package

[near hazel]

👋 Need Help https://github.com/dagger/dagger/issues/9471

[junior girder]

Replied in the issue 🙂

[stark dagger]

@junior girder @near hazel random suggestion that another option could be to use a pure python git implementation like https://github.com/jelmer/dulwich ?

[junior girder]

That’s why I asked if there were more intended uses for git in python. Otherwise it doesn’t seem worth it. There’s also the core git API call

[near hazel]

That’s why I asked if there were more

[sweet nimbus]

Hey all,

I am creating a core standard Dagger module that contains most of the common pipeline steps and other standard CI/CD scripts. I am trying to better organize the module which will likely be fairly substantial, and I was thinking that it would be nice to have the functions logically spread across different files/folders. Is there a way that is idiomatic to Dagger to do this?

For reference we are using Python (with the hatchling builder).

[junior girder]

Yeah, just group related functions in their own types and files. See an example: https://github.com/dagger/dagger/tree/main/sdk/python/dev/src/python_sdk_dev

[sweet nimbus]

Hey Helder,

So what I was thinking was more so a way to build up the main class (PythonSdkDev class in your example) as if all of the functions are in this main class without calling them in the same file. So then I could have multiple files that build up the main class, but I don't have to redefine them or call them in the main.py file again. There are ways to do this in Python, I was just wondering if there was already a more dagger specific or recommended way.

Thanks for the feedback by the way!

[junior girder]

Module split

[pseudo folio]

Hi all,

I am new to developing with dagger and looking to implement unit testing for a pre-existing dagger module written in python. What are the current best practices for unit testing dagger modules, i.e. can a test framework such as pytest be utilized or is there another process entirely?

[mortal sapphire]

Is anyone interested in working on a dager-dagger or dagger-dagster integration?

Some use-cases I think it might be useful for:

• simply calling Dagger functions and builds from Dagster jobs. This would require translating Dagger rich types like Directory or Container to Dagster's config type system and making an IOManager for them
• the opposite: using Dagster's orchestration capabilities inside Dagger functions to easily express cross-step dependencies (currently I'm hacking this together by douing synchronization with shared anyio.Event ). This would require figuring out good APIs to chain Dagger calls within a single Dagger function and across separate Dagger function invocations

[ionic hare]

Can you give an example of what you mean by "cross-step dependency"? I'm having trouble picturing it.

I would be very surprised if you needed a whole separate orchestration framework unless you're doing something very unusual, or very specific to that framework. But very possible that I'm missing something!

[hasty raven]

I couldn't find any information about a bug with repositories that have one letter between the dashes.

If you are experiencing an error like main object not found and your Dagger module is named like what-a-bummer, you may want to try the following steps:

1. Rename the package in dagger.json, pyproject.toml from what-a-bummer to something like what-abummer
2. Rename the main class from WhatABummer to WhatAbummer

[crude frigate]

Is it possible to have a pydantic list of enum using the python sdk ?

[junior girder]

There's no specific pydantic integration, but it can be added. Can you give an example of what you want to do?

[scarlet sinew]

import dagger
from dagger import dag, function, object_type, enum_type
from typing import List


@enum_type
class VisitReason(dagger.Enum):
    HOSPITAL = "hospital"
    DOCTOR = "doctor"
    CHIROPRACTOR = "chiropractor"
    DENTIST = "dentist"
    DENTAL_CLINIC = "dental_clinic"
    MEDICAL_LAB = "medical_lab"


@object_type
class DaggerDemo:
    @function
    def place_search(self, visit_reasons: List[VisitReason], loc_arg: str, rad_arg: str) -> str:
        return ""```

Error

[junior girder]

Don't use typing.List, that's deprecated. Just use the builtin list.

[scarlet sinew]

what version of python is dagger based on?

[junior girder]

Python 3.10+ is supported. The default is 3.12, but you can pin another.

[glacial lance]

Hey! I'm having troubled upgrading our pipelines to dagger engine to 1.7.0. This is a behind the VPN setup, with Athens GOPROXY being a required way to download anything. When trying to run existing pipelines which worked with 0.15.3 the setup of pipeline module fails with o: download go1.24.0 for darwin/arm64: toolchain not available

Now our pipelines are written in python and I noticed the go toolchain is different than the one provided by the go image. If I understand the code correctly this is where this stuff is set:

• Go version in go-alpine container: https://github.com/dagger/dagger/blob/v0.17.0/engine/distconsts/consts.go?plain=1#L30
• Go version and toolchain used by python sdk: https://github.com/dagger/dagger/blob/v0.17.0/sdk/python/runtime/go.mod#L3-L5

Is there a reason 1.24.0 toolchain is set there? Could this be a what go mod tidy added and it was copied from the developers machine?

This also adds additional slowdown for dagger engine startup, since toolchain will have to be downloaded. But if you are inside a more restricted environment you will be hit with this potential optimisations.

But maybe I'm wrong and there is a feature of go 1.24.0 being used somewhere in the python sdk.

[junior girder]

Go Toolchain

[fierce ermine]

Hi there. Nice to meet you all. I am currently tinkering around with dagger with ai agent. I have everything set up, and can see the agent change my code, as well as able to verify the changes. But I cant get that changes out from dagger system, despite calling export("."). Can anyone help me on this? did I missed any documentation around this?

[fierce ermine]

nvm, seems like I got the answer here #1356372041133199411 message

[sullen frost]

@junior girder I was translating a Python (pydantic, OpenAI libs, etc) agentic workflow into Dagger LLM style from https://learn.deeplearning.ai/courses/evaluating-ai-agents/lesson/pag5y/lab-1:-building-your-agent

https://github.com/jpadams/data-analyst

It looks like they are giving an object to the LLM as a desired output shape like we might do with with_objecttype_output, but I'm not sure how to address this bit in a way that gives me that kind of automatic binding. I thought about putting this sort of object with fields in a separate module and installing it, but I got complaints from dagger...maybe due to syntax, maybe due to having no functions in the module. Any ideas?

# class defining the response format of step 1 of tool 3
class VisualizationConfig(BaseModel):
    chart_type: str = Field(..., description="Type of chart to generate")
    x_axis: str = Field(..., description="Name of the x-axis column")
    y_axis: str = Field(..., description="Name of the y-axis column")
    title: str = Field(..., description="Title of the chart")


# In[ ]:


# code for step 1 of tool 3
def extract_chart_config(data: str, visualization_goal: str) -> dict:
    """Generate chart visualization configuration
    
    Args:
        data: String containing the data to visualize
        visualization_goal: Description of what the visualization should show
        
    Returns:
        Dictionary containing line chart configuration
    """
    formatted_prompt = CHART_CONFIGURATION_PROMPT.format(data=data,
                                                         visualization_goal=visualization_goal)
    
    response = client.beta.chat.completions.parse(
        model=MODEL,
        messages=[{"role": "user", "content": formatted_prompt}],
        response_format=VisualizationConfig,
    )
    
    try:
        # Extract axis and title info from response
        content = response.choices[0].message.content
        
        # Return structured chart config
        return {
            "chart_type": content.chart_type,
            "x_axis": content.x_axis,
            "y_axis": content.y_axis,
            "title": content.title,
            "data": data
        }
    except Exception:
        return {
            "chart_type": "line", 
            "x_axis": "date",
            "y_axis": "value",
            "title": visualization_goal,
            "data": data
        }

[junior girder]

Evaluating AI Agents - DeepLearning.AI

[idle mountain]

Hello everyone. 👋 First post here!
I have been using Dagger 0.13.x. Did not touch dagger.io for some time until today!
I discovered the new web site, and the domain oriented communication (AI agent, CI etc ...) which makes things clearer than before I think. So kudos to everyone who worked on this!

Now to what brings me here 😆 . It really seems to be something wrong with my setup (who was working flawlessly under 0.13.X though ...)

After running those 2 simple steps :

• dagger init --sdk-python
• dagger install <watevermodule>

Runnining a simple

def func() -> dag.Helm:
   return (
     dag.helm()
   )

Would fail, stating that 'Client' object has no attribute "Helm" (if the module is Helm for instance)
It feels like the code generation is not done properly (while there are no evidence of this)
I am using the latest version of dagger and sdk .. I have been uninstalling (the whole thing with Images, volumes etc ...) and reinstalling stuff from scratch .. and feel a little clueless right now .. Happy if anyone would have pointers. 🙏 !

[junior girder]

Hello everyone. 👋 First post here!

[steep widget]

I have a pipeline that started failing once upgraded to 18.4

process "uv pip install -e ./sdk -e . --no-deps -r requirements.lock" did not complete successfully: exit code: 2

Using Python 3.12.10 environment at: /usr/local
error: Distribution not found at: file:///src/xxh3:80368b2c0173d09e/plausible/dagger/sdk

Any ideas? Nothing changed except for the dagger version.

[junior girder]

I have a pipeline that started failing

[idle mountain]

Hello 👋

I am stuggling in trying to publish to registry.gitlab.com
This is the code I am using locally from my laptop👇

            dag.container()
            .build(context=syncer_folder)
            .with_registry_auth(**url**,"username", **token**)
            .publish(**url**)

The url I am using is of the form: registry.gitlab.com/<username>/<path>/<imagename>

The token I am using is working, locally (I can docker login and docker publish using the token).

This is the error I am getting
👇

│ ! https://gitlab.com/jwt/auth: 404 Not Found

I have been trying to use with or without the username in the url and used different type of tokens (group, project, deploy token) but with no difference. Actualy I am not even sure which this "username" is 🙄 referring to, the name of the token, or the string right after "gitlab.com" URL?

Also it seems like the "image" path is being created in the registry (ie: I can see it in the registry UI) but the corresponding image tag not.

Depending on which username I am using , I have also this message

server message: insufficient_scope: authorization failed

while the scopes are all checked.

Anyone who made that work specifically with a gitlab registry (otherwise I start to feel there is a bug here 🤔 ) ?

Thx!

[junior girder]

Publish to private gitlab

[junior girder]

Hey <@&1113692108755308593>! There's been a regression in v0.18.4 for Python SDK users that haven't migrated to uv yet in some modules (i.e., no uv.lock). Those modules won't vendor the dagger-io client library properly in v0.18.4 but there's a fix and mitigation instructions in https://github.com/dagger/dagger/pull/10252.

[south geode]

@junior girder Is it possible to store custom types as values for another object-type? Something like:

@object_type
class Test:
        name: str = field()

@object_type
class Local: # this is the actual module
        tests: list[Test] = field()

        @function
        def list_tests(self) -> str:
                return ", ".join([f"{test.name}" for test in self.tests])

I've tried this and I'm getting "failed to convert result to primitive values, invalid value for type, expected an iterable". The only types used are str, int, dagger.Container

[junior girder]

We try to keep consistency in all SDKs officially supported by the Dagger team so I wouldn't worry about update frequency or coverage. I suggest you pick the language that you or the module maintainers are more comfortable with.

You can also create multiple modules for different components of your project and those can be in different languages if you have different teams with different preferences, for example a Python backend module and a TypeScript frontend module.

[waxen scarab]

@junior girder ran into this issue on the livestream on v0.18.5 yesterday - here is the livestream at the timestamp for reference: https://www.youtube.com/live/fJyE939YcqY?feature=shared&t=530

[ionic hare]

I would like to develop a Dagger module in python without installing any python tool on my local machine, including uv. Can I do that?

[maiden plank]

I think that depends if you want LSP support in your IDE. If no, no tools required. If yes, I think every language requires you to install something other than dagger

[ionic hare]

I see 'uv' and 'uv sync' being mentioned a lot. Is it really true that I only need to install tools for LSP?

In my mind LSP support is solved by clicking "yes" when my IDE detects the language and asks if I want to install the IDE plugin. I shouldn't have to install or learn any other python-specific tool ever. I just want to make sure that remains true. Otherwise it's a slippery slope

[waxen scarab]

With Zed, most languages work well out of the box.. like PHP. I have not found the same with python and Zed though

[maiden plank]

Could use another set of eyes, I can't figure out why this constructor is incorrect 🤔

@object_type
class Workspace:
    """A module for editing code"""
    source: dagger.Directory

    @classmethod
    def create(
        cls,
        source: Annotated[dagger.Directory, Doc("The source directory")]
    ):
        return cls(source=source)

│ ! 'Workspace' object has no attribute 'source'
! input: helloDagger.develop 'Workspace' object has no attribute 'source'

[waxen scarab]

self needs to be passed to the func?

[maiden plank]

no, it's a constructor. But it turns out I actually don't need a constructor in this case because it's the same as the auto __init__

[junior girder]

Yes, if your IDE doesn't support uv natively I bet it will soon enough. uv is a relatively new tool but it can replace all the others so you don't even need to install Python, you only need uv which is a single compiled binary that's easy to install (download to $PATH) and also really fast when running. So it's popularity keeps growing exponentially and support for it in IDEs (and other tools like dependabot) is growing too. In that case you don't need to install it yourself, you can let the IDE handle it.

[junior girder]

I think you just have an older uv version locally, so you just need to update it: uv self update. It's explained in the issue you linked to. No need to edit the uv.lock manually.

[junior girder]

As for the LSP troubleshooting I'm seeing in that livestream, IDEs tend to only consider a .venv at the root of a project. Since your Python module is in a subdir (.dagger/.venv) I would just open a new window like a second project from that path. It's the same in Go (at least with my editor).

In some IDEs like PyCharm you can also just right click a directory and set it as another sources directory, but it's not obvious for people unfamiliar with Python or how their IDE works with Python. But I cringe when I see people searching the web and seeing a solution like editing a json config to make LSP work. It makes it seem like it's harder than it needs to be.

I want to improve workspace support for monorepos just haven't had time for that yet. It won't be the default though, but will easily allow you to have a single virtual env for both app and Dagger module at the root of the whole project.

The dead simple solution for non-Python developers is to just open a window with the module at the root of the IDE workspace. \cc @maiden plank @sullen frost

EDIT: to be clear, it is possible to have both on the same window but you have to check the IDE's documentation on how to setup a Python monorepo with separate virtual environments. Nothing special about a Dagger module, it's just a normal Python package.

[junior girder]

Yeah, you don't need the create factory class method here. It was added to support making async calls during object construction and is only called automatically for the module's main object so it's not standard Python, it's specific to Dagger. But I'm curious because there isn't anything wrong with that code so wondering if it was an issue with the imports or something.

[sullen frost]

As for the LSP troubleshooting I'm

[ionic hare]

Slippery slope imo.

Why do I need to call uv sync after dagger develop? Why can't dagger develop call uv for me in the containerized environment?

[junior girder]

Because of the way virtual envs work, they're not portable. When you do uv sync locally the virtual env is created with paths that are specific to your host which avoids a lot of issues than trying to do this in an unsupported way.

[ionic hare]

Ah I see

Can't wait for a native support for dev environments in Dagger 🙂 So I never have to develop anything on my host machine ever (dagger modules or otherwise)

[waxen scarab]

https://allthingsopen.org/articles/comparing-uv-and-pip-for-faster-python-package-management

[south geode]

Astral are more openly talking about Ty now as well, they're doing great work

[silver warren]

Hi, I upgrade my dagger engine on 0.18.6 from 0.16.x but now I have this error: ! unhashable type: 'list' but I don't understand from where it could come

I did a dagger develop

[junior girder]

I did a dagger develop

[junior girder]

Hi, I upgrade my dagger engine on 0.18.6

[fringe relic]

hello!

I am stuck on this.
I need to generate YAML in my python function and then I need to load that YAML in a container as a file and execute helm command on it.

How can i load string/YAML into a container as a FILE object?

[fringe relic]

I found the solution.

    @function
    async def test_stuff(self) -> str: 
        helm = self.test_container()
        f = dag.directory().with_new_file("/foo.txt", "Hello World")
        await helm.with_directory("/foo", f).terminal()
        return ""

I have confirmed that the files exists at path "/foo" in the container.

[bleak junco]

What's the difference between the SDK used to develop a dagger module and the open SDK installed using pip install dagger-io?
Where is the doc for the module one, please?

[ionic hare]

We developed the bad habit of calling "SDK" two related but different things, sorry...

• pip install dagger-io installs our python client library
• The Dagger Python SDK is a development kit which includes the client library, alongside other tools

Here's how the docs used to explain it:

Dagger SDKs provide resources for developing Dagger modules using a familiar language and toolchain. Each SDK provides:

1. A client generator, to consume the Dagger API with native code.
2. A server generator, to extend the Dagger API with native code.
3. Examples and reference documentation

[bleak junco]

Would you know why the provisioning directory doesn't get copied when executing dagger develop --sdk python?

[ionic hare]

Sorry, I don't have enough in-depth knowledge of the Python SDK to know. But perhaps @grave sinew @civic storm can help?

[grave sinew]

That's a good question!

It's because in the context of a module, you call it using the dagger CLI (dagger call or dagger shell), so the engine is automatically provisioned by the CLI. That means the SDK when called in that context, will never have to provision the engine by itself so we simply remove that part of the code since it will be never used.

However, when using the public SDK library, you may call your script through dagger run (which will provision an engine) or you may use python myscript.py. In that specific case, the SDK will automatically provision an engine (the same way the CLI does)

[bleak junco]

Ok, that's a good answer. Thanks!

So this:

import dagger
import sys
from dagger import dag
import asyncio

async def terraform_workflow():
    cfg = dagger.Config(log_output=sys.stderr)

    # Initialize Dagger client
    async with dagger.connection(cfg):
        # Set up the container with Terraform installed
        terraform = (
            dag.container()
            .from_("hashicorp/terraform:latest")
            .with_workdir("/terraform")
        )
        
        # Mount the current directory (assuming it contains Terraform files)
        terraform = terraform.with_directory(
            "/terraform",
            dag.host().directory("."),
        )
        
        # Execute terraform init
        print("Running terraform init...")
        init = await terraform.with_exec(["init"]).stdout()
        print(init)
        
        # Execute terraform plan
        print("\nRunning terraform plan...")
        plan = await terraform.with_exec(["plan"]).stdout()
        print(plan)
        
        # Pause for user input
        print("\nReview the plan above. Do you want to proceed with apply?")
        user_input = input("Type 'yes' to continue or any other key to abort: ")
        
        if user_input.lower() == 'yes':
            print("Proceeding with terraform apply...")
            # apply = await terraform.with_exec(["apply", "-auto-approve"]).stdout()
            # print(apply)
            print("Apply would execute here (commented out for safety)")
        else:
            print("Execution aborted by user.")

# Run the async function
asyncio.run(terraform_workflow())

Doesn't make sense as a module. And that's why the Config and connection are not present...

[grave sinew]

Exactly, what you are showing here looks like a standalone dagger script, not a dagger module 🙂
You can convert it to a module indeed, but the code will be slightly different

[bleak junco]

Using @functions, etc... Ok, got you. Thanks.

[grave sinew]

No problem! You can check https://docs.dagger.io/api/custom-functions for more details about modules

[short crow]

https://github.com/dagger/dagger/issues/10746

[vague slate]

Hi, is python sdk support interface type?

[junior girder]

Yes: https://docs.dagger.io/api/interfaces

[vague slate]

Oh, Nice! Thank you @junior girder 🙏

[ionic hare]

Could we use this to create a single-file python module? 🙂 https://docs.astral.sh/uv/guides/scripts/#declaring-script-dependencies

[junior girder]

Yeah, certainly. Planning on making an example but not make it a default unless we add support for init’ing from different templates via some flag.

[ionic hare]

Is there a downside to that pattern over others? (other than familiarity). For example are there dependencies that just won't work that way?

[junior girder]

The problem is it’s more complicated if you want to scale. It’s also more complicated to downscale to a simple script. Both require specific instructions (to and from a script). I’d rather err on the side of the full package because it’s simpler to scale and not that bad for the simple cases. But if we had multiple templates for sure I’d have a simple script vs full package options.

[junior girder]

When I say it’s more complicated to scale what I mean is that I’d expect to get lots of support requests from people but we can always answer with a link to the docs.

[junior girder]

I got it to work, here's the full file tree for a module:

.
├── .gitignore
├── .gitattributes
├── dagger.json
├── dagger_gen.py
├── main.py
└── main.py.lock

And this is what a minimal main.py looks like:

# /// script
# requires-python = ">=3.13"
# dependencies = [
#     "dagger-io",
# ]
# ///
import dagger
from dagger.mod import run


@dagger.object_type
class MyModule:
    @dagger.function
    def echo(self, msg: str) -> str:
        return msg


run(MyModule)

So basically:

• src/<module_name>/__init__.py is deleted
• src/<module_name>/main.py moves to main.py
• pyproject.toml is incorporated into main.py thus no longer necessary
• There’s two extra boilerplate lines for run()
• uv.lock is renamed to main.py.lock (it’s how uv works)
• the whole vendored lib under sdk/ gets replaced by a single dagger_gen.py with just the client bindings (but only if not in a dev build of the engine)
• you can import other .py files next to main.py so you can still split functions into multiple files if you want

[slim badge]

This would be a really nice DX improvement for python modules!

[slow dagger]

Hello, I have opened this issue: https://github.com/dagger/dagger/issues/10885

I think there is a breaking change with the gql library. They had released a 4.0.0 version and its throwing errors when trying to connect to the dagger engine. I am using the python-sdk==0.16.1

[willow sail]

already responded on github, but just an fyi, the idea is to ship a v0.18.15 today to fix this 🙏

[steep widget]

Hey! I have a previously working module breaking in 18.5 & 18.6 is this a known issue?

Using Python 3.13.5 environment at: /usr/local
Resolved 46 packages in 43ms
   Building lxml==5.2.2
   Building dagger-io @ file:///src/xxh3:86839813ecd59e22/nostalgia/dagger/sdk
   Building main @ file:///src/xxh3:86839813ecd59e22/nostalgia/dagger
      Built dagger-io @ file:///src/xxh3:86839813ecd59e22/nostalgia/dagger/sdk
      Built main @ file:///src/xxh3:86839813ecd59e22/nostalgia/dagger
  × Failed to build `lxml==5.2.2`
  ├─▶ The build backend returned an error
  ╰─▶ Call to `setuptools.build_meta:__legacy__.build_wheel` failed (exit
      status: 1)

      [stdout]
      Building lxml version 5.2.2.
      Building without Cython.
      Error: Please make sure the libxml2 and libxslt development packages
      are installed.

      [stderr]
      <string>:67: UserWarning: pkg_resources is deprecated as an API.
      See https://setuptools.pypa.io/en/latest/pkg_resources.html. The
      pkg_resources package is slated for removal as early as 2025-11-30.
      Refrain from using this package or pin to Setuptools<81.

      hint: This usually indicates a problem with the package or the build
      environment.

[steep widget]

Hey! I have a previously working module

[hasty raven]

Hello

Is there a way to generate python dagger sdk from within Dagger functions?

Context:
I would like to implement a function test_dagger that will launch uv run pytest for dagger pytest tests.
But I need dagger sdk package to be available inside my test container.
I can try to use sdk package from pypi but I had some problems with this approach in the past.

What will be a right approach for this?

[junior girder]

generate python dagger sdk from within Dagger functions?

[blissful scaffold]

I was scrolling this channel and came across this. Is this already available or is it just some local PoC by you and not released anywhere yet?

[thorny patrol]

I'm currently doing something like this to make it work:

pip install dagger-io==0.15.1 && \
pip install gql==3.5.3 && \
echo "done"

[crude frigate]

Quick question on the Python SDK. I'm dogfooding how we'll expose to developers the ability to deprecate their module's types / fields / everything.

Helder mentioned:

Yes, it's doable. We can have a @dagger.deprecated("reason") decorator for that. Or just @dagger.function(deprecated="reason").

Right now, I am set on:

  import enum
  from typing_extensions import Annotated

  import dagger
  from dagger import Deprecated, enum_type, field, function, object_type


  @object_type(deprecated="This module is deprecated and will be removed in future versions.")
  class Test:
      @field(deprecated="This field is deprecated and will be removed in future versions.")
      legacy_field: str = ""

      @function()
      async def echo_string(
          self,
          input: Annotated[str, Deprecated("Use 'other' instead of 'input'.")],
          other: str,
      ) -> str:
          return input

      @function(deprecated="Prefer echo_string instead.")
      async def legacy_summarize(self, note: str) -> "LegacyRecord":
          return LegacyRecord(note=note)

      @function()
      def use_mode(self, mode: "Mode") -> "Mode":
          return mode


  @object_type(deprecated="This type is deprecated and kept only for retro-compatibility.")
  class LegacyRecord:
      note: Annotated[str, Deprecated("This field is deprecated and will be removed in future versions.")]


  @enum_type
  class Mode(enum.Enum):
      """.. deprecated:: Mode is deprecated; use Zeta instead."""

      Alpha = "alpha"
      """.. deprecated:: alpha is deprecated; use Zeta instead."""

      Beta = "beta"
      """.. deprecated:: beta is deprecated; use Zeta instead."""

      Zeta = "zeta"

But my question is on enums ... I can't seem to have an @deprecated on enum members, so either I rely on the docstring OR we change the way users assign enum members Alpha = enum_member("alpha", deprecated="alpha is deprecated; use Zeta instead.")

[ionic hare]

@tropic holly we would love your thoughts on this... 👆

We have an unofficial policy of aligning with uv and its ecosystem, whenever there is a need to choose a DX convention and the Python ecosystem is too fragmented to give us a single answer.

In this case, to our knowledge uv et al doesn't have a convention for the very specific problem of annotating an enum value... (for the purpose of deprecating it). But if you personally have an opinion, we would prefer to align with it.

TLDR: "what would Charlie do?" 🙂

[crude frigate]

@civic storm what do you think ? Seems like a sensible UX. I don't like the fragmentation just for enum values, but it's kind not too bad and is a standard too

[civic storm]

I think decorators are good. Regarding enums... Enums are always something weird, in almost all languages. So if enums are not working the same way, I'm not that surprised. I don't know enough good practices in Python to have a strong opinion on which one is best, but I think I prefer to have the docstring than something like enum_member(value, args), mostly because deprecated is not at the same level than the value itself.

[gray perch]

hello, looking for advise on how a shareable and reusable dagger modules can be organized in a github repo to be used in other repos

[left stag]

hello, looking for advise on how a

[granite anchor]

Python newbie here... I want to require arguments at the module level so that all other member functions can make use of them via self. Below is my first idea of how to do this but I can't seem to get it too work if I put extra arguments on the __init__ method. Is this a valid approach, and if so what am i doing wrong?

from dagger import function, object_type

@object_type
class Daggerplay:

    @function
    def __init__(self,msg: str):

        self.msg = msg

    @function
    async def saymsg(self) -> str:
        """Test"""
        return f"msg [{self.msg}]"

here is the result:

$ dagger call --msg="xxx" saymsg
✔ connect 0.3s
✔ load module: . 10.3s
✔ parsing command line arguments 0.0s

✔ daggerplay(msg: "xxx"): Daggerplay! 3.8s
✘ .saymsg: String! 3.7s ERROR
! InvalidInputError: Failed to instantiate parent object 'object 'daggerplay.main.Daggerplay'': invalid type (Daggerplay.__init__() missing 1 required
  positional argument: 'msg') @ daggerplay.main.Daggerplay
  This could be an error in the Python SDK. If so, please file a bug report.

[late hornet]

In the docs it looks like its defined a bit differently

from dagger import function, object_type


@object_type
class MyModule:
    greeting: str = "Hello"
    name: str = "World"

    @function
    def message(self) -> str:
        return f"{self.greeting}, {self.name}!"

https://docs.dagger.io/extending/constructors

scrolling down, looks like it goes into a bit more detail

hmm, and it looks like it can be even more extended with a special create function if async is needed

Yeha, i just did a mini local dagger init --sdk=python and tried to repo. If you remove the init and then define the fields at the class level that should work for you

[granite anchor]

Thanks for the reply... I realize that I can set fields at the class level and remove the init() but in my case I want to accept a source directory (aka a dynamic value set when the function is called) and for that I don't know how to do that at the class level, I can't just hard code it. The only way I know how to accept a host directory is from the command line.

[granite anchor]

I found what I needed,
https://docs.dagger.io/cookbook/filesystems/#set-a-module-wide-default-path

I can set it on the class like this:

@object_type
class MyModule:
    source: Annotated[dagger.Directory, DefaultPath(".")]

Thanks

[idle mountain]

Hello everyone 👋

I am trying to retrieve a secret with with_secret_variable then reuse the value of this variable in another variable using with_env_variable with expand=True but it seems like this is not working (the variable is expanded but to an empty string.

In other words, considering this snippet 👇

    .with_secret_variable("TOKEN1",token)
.with_env_variable("CONCATVAR","VALUEIS:${TOKEN1}",expand=True)

CONCATVAR's value would be : "VALUEIS:"

Is it how it is supposed to (not) work, are there other ways to handle this use case? Could it be a bug?

Thx!

[ionic hare]

it's possible that expand does not include secrets in the expansion, either intentionally or as a bug.

Can you confirm that the expansion works if you change to a regular with_env_variable?

[floral harness]

Hi, a quick chat with @ionic hare yesterday in Tel Aviv, since I love writing pytest pluggins, had me thinking and I sketch the following idea for a plugin: https://docs.google.com/document/d/1ydNGnMFFdEOYyOCx1u9gvzC049Syq-oux4S2tgcOtew/edit?usp=drivesdk

It's mostly me a Gemini vibing
(I heard of dagger for the first time yesterday, so be gentle with me 🙂 )

[floral harness]

I'm migrating a large Bash script to Dagger and ran into a UX issue. While we can set default values for File/Directory types via annotations, we cannot do the same for Secret or Socket arguments.

This causes friction in two main areas:

• AWS Auth: I can't default to standard paths like ~/.aws/*.
• Docker: Users must manually pass the Unix socket every time.

This forces users to type out long, cumbersome commands for basic functionality, making the migration from Bash feel less user-friendly. Are there plans to support defaults for these types? or any ideas to go around those things ?

[floral harness]

Missing default value support for Secrets and Sockets

[civic storm]

We just published a Dagger toolchain to run python tests and automatically send telemetry traces. It means it's now possible to see all the tests that have been run in the TUI/Dagger Cloud.
https://github.com/dagger/pytest

The repository contains a quick example on how to use it on an existing project (tested on external real projects).
Basically the way to use it is to install the toolchain dagger toolchain install github.com/dagger/pytest then run dagger check pytest and that's it!

I haven't tested it with all kind of combination, so if it's not working on some project do not hesitate to reach out.

[hushed snow]

So, any time I pass in a parameter to a nested object type I get an exception complaining about missing positional arguments. Is there a trick to this? This was enough to make it error out instead of being able to invoke the functions. In theory this should be able to do dagger call test --param test run_echo no?

import dagger
from dagger import dag, function, object_type


@object_type
class TestClass:
    def __init__(self, param: str):
        self.param = param

    @function
    def run_echo(self) -> dagger.Container:
      return dag.container().from_("alpine:latest").with_exec(["echo", self.param])

@object_type
class DaggerTest2:
    @function
    async def test(self, param: str) -> TestClass:
        return TestClass(param)

dagger call test --param thing run-echo
✔ connect 0.2s
✔ load module: . 0.3s

✔ daggerTest2: DaggerTest2! 0.7s
✔ .test(param: "thing"): DaggerTest2TestClass! 0.7s
✘ .runEcho: Container! 0.7s ERROR
! InvalidInputError: Failed to instantiate parent object 'object 'dagger_test_2.main.TestClass'': invalid type (TestClass.__init__() missing 1 required positional argument:
  'param') @ dagger_test_2.main.TestClass
  This could be an error in the Python SDK. If so, please file a bug report.

✔ parsing command line arguments 0.0s

[civic storm]

You have to define the field in the TestClass object:

@object_type
class TestClass:
    param: str

    def __init__(self, param: str):
        self.param = param

That way the field is known, and will be part of the serialization. In short without it Dagger doesn't know the field exist, the value can't be stored and retrieved.
With that change:

$ dagger call test --param thing run-echo stdout
✔ connect 0.2s
✔ load module: . 0.3s
✔ parsing command line arguments 0.0s

✔ daggerTest2: DaggerTest2! 0.0s
✔ .test(param: "thing"): DaggerTest2TestClass! 0.0s
✔ .runEcho: Container! 0.1s
✔ .stdout: String! 0.1s

thing

[hushed snow]

Apparently I managed to miss that incantation in my guess and check. thank you very much. I love this tool, it's glorious.

[hushed snow]

Is there a way to shove those parameters into an object that is a Pydantic BaseModel or is that level of integration not a thing?

For example: I'm putting some configuration options into pyproject.toml. I have defaults but I want to allow overrides.

[tool.daggerTest2.plugins.terraform]
enabled = true
configurationPath = "iac"

So what I do is in daggerTest2 I load the pyproject.toml and pass it down into the plugin class:

@function
    async def terraform(self, config_file: Annotated[File, DefaultPath("pyproject.toml")]) -> TerraformPlugin:
        return TerraformPlugin(await dagger_test.config.get_project_config(config_file))

However, that plugin is a Pydantic object so I can go TerraformConfig.model_validate(json).

That property of course is now no longer a string and is an object. If I do an object_type I see I can get it to work, but I can't use pydantic to validate it then because it's missing pydantic properties. (i.e. I can't extend BaseModel). I can see maybe I can abuse cattrs to get into my structure hook, but my google results for that have been lacking so far (I'm assuming that's one way if I can find the dagger converter).

! failed to call module "dagger-test" to get functions: ModuleLoadError: Unsupported type: <class 'dagger_test.terraform.main.Terraform'>. Register a structure hook for it.

I can of course work around it by just passing the full pyproject file around and grabbing what I want later so it's not the end of the world, just seems like it'd be handy to be able to delegate the parsing of a section of it to the plugin that defines the format of the section without requiring them to know how to parse whatever file I shove it in.

[hushed snow]

Ah hah! I found how I missed the incantation. The thing in the constructor appears to need to be the same name. That explains a lot. So I can swap types, but I can't swap the name between the property and the constructor.

[wraith matrix]

We're using Dagger within a Python custom application and are having trouble updating from Python 3.13.12 to 3.14.3: import dagger leads to an error, see the attached stack trace. Is beartype compatible with Python 3.14?

[civic storm]

Thanks for the report. Yes, it looks like there's an issue between the minimum beartype version the SDK requires and python 3.14.
While I'm looking into bumping python and the dependencies, could you try to work around by upgrading beartype to version 0.22.0?
My guess is it should unblock you for now.

[civic storm]

this looks like to be this issue: https://github.com/beartype/beartype/issues/555

[wraith matrix]

Indeed, updating to beartype 0.22.0 seems to have done the trick! Thanks so much! Maybe you could add a line to the Dagger SDK's pyproject.toml, requiring beartype >= 0.22.0 when python >= 3.14?

[civic storm]

Yes, I'm preparing a PR to bump python and some dependencies, that should solve it the right way.
Happy it fixes the issue for now 🙂

[tall cairn]

Got a question abot dagger_gen.py what is it and is it needed in remote git repo or can we put it in gitignore

[civic storm]

Dagger modules are based on a code generation phase. That allows to have a dynamic schema on the engine side, and a library generated for each language. That way when you're writing a module in python, it feels native even if some parts are made in a different language (for instance to use a module written in Go as a dependency of a module written in python)
Files like dagger_gen.py contains this code generation.
You should be safe to remove it/ignore it, this file should be re-created at runtime anyway (to ensure we are always calling functions with the right codegen file.

[tall cairn]

Thanks chief 💪

[silver warren]

Hi, we are using different dagger modules in our repo, these modules are written in python but few month ago we analyse our pipelines and observe a long time in the sdk initialization (thread here: https://discordapp.com/channels/707636530424053791/1433826641250877562/1433826641250877562 at the beginning I thought it was an issue on the connection but after a call and running some commands the real issue was on the sdk initialization). We still have the issue and now it's real big part of the time in our pipelines, there is a way to optimize that ?

[ionic hare]

Hello! The first step is to upgrade, we've made many speed optimizations since 0.19.0.

I saw in another thread that you're getting issues upgrading, so assuming that you haven't yet evaluated python sdk init on latest release?

[copper orchid]

The CLI is not respecting the default enum value argument with the python SDK?

[mortal sapphire]

Is there a reason I cannot run codegen (to produce the typically gitignored sdk directory) for a Python Dagger module without all of these files?

generated = (
    dag.directory()
    .with_file("dagger.json", pkg_dir.file("dagger.json"))
    .with_file("pyproject.toml", pkg_dir.file("pyproject.toml"))
    .with_file("uv.lock", pkg_dir.file("uv.lock"))
    .with_new_file(f"src/{package_dir}/__init__.py", "")
    .as_module_source()
    .generated_context_directory()
)

In practice this fails if any of these are not provided, but actually only dagger.json is relevant (should be required) to codegen, isn't it? Is this a bug?

---

Hmm there is something more going on. Didn't actually get it working yet. But it's weird to me why dagger.json is not enough. I don't think my original assumption for what causes my problems was correct.

What I'm trying to do is running uv sync inside a Dagger build for a Dagger module. It depends on ./sdk, but it is gitignored, so I'm trying to re-generate it. How can I achieve that?

[civic storm]

I'll have a look. But that doesn't really surprised me. Those files are what is expected to represent a python module.
Just to understand, what is the issue by passing the actual module source dir to the function?

Things will change soon on this area, the python codegen will generate more code, but remove all codegen at runtime.

[mortal sapphire]

I want to avoid invalidating the build cache too early. I only want to rerun codegen when dagger.json changes.

[civic storm]

I'm trying with dagger shell, but that looks like to work on my side:

$ dagger -M
▶ directory | with-new-file "dagger.json" '{"name": "test","engineVersion": "v0.20.6","sdk": {"source": "python"}}' | as-module-source | generated-context-directory | directory "sdk" | entries
LICENSE
README.md
codegen/
pyproject.toml
src/
uv.lock

[mortal sapphire]

Oh, thanks a lot!

I think my problem was using the original dagger.File for dagger.json, while you created a fresh one.
I was able to achieve the same result by reading the contents of the original file and making a new file with these contents.

I'm not sure exactly why didn't my original aproach work, something about Dagger tracking some metadata from the original dagger.json context?

[civic storm]

I embedded the dagger.json as a way to make it easier to reproduce, but that could be interesting to see what's inside. Does the module has dependencies for instance?

[mortal sapphire]

The "main" (repo root) module - which triggers the build for the inner module with a Dagger check - depends on the inner module, while the inner module (the one being built) doesn't have any dependencies

[mortal sapphire]

https://github.com/typesafe-ai/daggerverse/tree/main

You'll find both modules here

[civic storm]

/tmp/tmp.LmZKk649oE/daggerverse $ dagger -M
▶ ✔ directory | with-file "dagger.json" $(host | file "dagger.json") | with-directory "uv-workspace" $(host | directory "uv-workspace") | as-module-source | generated-context-directory | directory ".dagger" | directory "sdk" | entries
LICENSE
README.md
codegen/
pyproject.toml
runtime/
src/
uv.lock

▶ ✔ directory | with-file "dagger.json" $(host | file "uv-workspace/dagger.json") | as-module-source | generated-context-directory | directory "sdk"
Directory@xxh3:85e0542e80801dd8

That seems to work in both cases with just the dagger.json. Of course from the main module you need to also add the dependency directory, as it's required to load the dependency and generate the bindings.

[mortal sapphire]

Since uv-workspace is a generic library module for building uv packages, consumed by other Dagger modules, it had to handle codegen internally. So running it from the outer module was not an option.

[civic storm]

I think I don't get it. If you cd uv-workspace and run

dagger -M -c 'directory | with-file "dagger.json" $(host | file "dagger.json") | as-module-source | generated-context-directory | directory "sdk" | export "sdk"'

Then you have the sdk directory exported.
My example above was from the root because I tried both, but the directory | with-file "dagger.json" $(host | file "uv-workspace/dagger.json") | as-module-source | generated-context-directory | directory "sdk" doesn't care where we are.
Or is there something else than just generating the sdk directory?

[mortal sapphire]

ok maybe I'll try that in the Python world again a bit later. I'm pretty sure it was failing for me (on Dagger Cloud NaticeCI only I think, but I am not sure anymore)

[civic storm]

Hi python 🐍 folks!
The recent release v0.20.7 introduced a new AST based analyzer that reads module's source code to extract the types, annotations, etc.
This will bring multiple benefits, including the support of self calls, but more widely this is one step towards multiple performance improvements.
But it looks like it contains some gaps 😥 Some types are not read as expected, some python syntaxes are not supported, etc.
Just to make it clear, the goal is to be a drop-in replacement, no change needed on the module's source code.

I'm currently building a way more extensive test suite so we can find the missing bits that are not yet good enough. It's happening in the PR #13095 (linked to the issue #10397
It already found and fix a bunch of cases, that's promising.

But as you all know, there's a lot of different ways to write something in python 😅
So I would be interested if any of you have open source python modules to share with me so I can include them in a test suite and ensure we are fully compatible so we can safely use the feature.

Thanks a lot in advance 🙏

[ionic hare]

cc <@&1113692108755308593> fans 👆

[pearl ridge]

Hello @civic storm ! I published this repository of public Dagger modules that is written entirely with Python SDK, if it can help 🙏
https://github.com/telchak/daggerverse

[civic storm]

Thanks! It does, I'm already using it in a test to compare the previous analyzer and the new AST one 🙂
https://github.com/dagger/dagger/pull/13095/changes#diff-1e3c672fc8930723108247ae8410d6700f2e8f0a19ed04729b0c3f87bddf44e4R3
That way it ensures both are giving the same output. And your repository already helped to catch some issues in the AST version (that have been fixed)

[hushed snow]

What's the magic incantation to add steps to the TUI? Is it calling another function decorated with '@function'? Trying to clean up my output from my build hierarchy and was expecting to condense the test of my package down into a single line 🤔