rust

[lunar vale]

https://discord.com/channels/707636530424053791/1140651088995942460

[somber owl]

Awesome

[modest wedge]

🦀

[lunar vale]

There is an experimental Rust SDK for scripting your CI in Rustlang 🙂 https://github.com/dagger/dagger/tree/main/sdk/rust

[desert temple]

Fyi rust will have a few upcoming breaking changes some internal and some for how it deals with types. This change overall is done to make including dagger rust in a library easier

[pure pier]

@desert temple Hey! Pinging as promised from Zoom - great demo! Here's a peek at what an interactive TUI + REPL could look like: https://asciinema.org/a/flETRxpizNQVuQai3TZ4SmLrv (taken from our Zenith dagger shell experiment)

I think this could work like dagger run -i <command that starts your REPL> - so instead of rendering the DAG, it would show the command full-screen like the linked recording. I'm curious if you have any suggestions, i.e. whether you want a more prominent display of the DAG somewhere, rather than having it all condensed to the status line at the bottom.

[dusky cave]

#1151407316088934471 message

[desert temple]

I've just released dagger-rust-sdk v0.3.2. I realize it has been a while. I am probably gonna see how I can set it up to your automatic releases. I am gonna chip away at it over the next month or so. (it is a bit of a pain to do manual releases, and I'd rather want something automatic), I can also see that the publish process is largely automatic, It already had the source generated without issue for 0.9.0 (which is honestly amazing as the codegen is a bit rough around the edges :,} ).

[desert temple]

Also if you see a sdk/rust issue on github, feel free to tag me, or something. I may not always see them as I don't monitor the issue log. (and github notifications suck)

[orchid osprey]

@desert temple (I opened the recent #5928). I just want to say a huge thank you for all your work on the SDK -- using Dagger from Rust is such a game changer. 🦀 😄

[desert temple]

No worries i will get to fixing some of the issues over the next week. If you've got any Ris or Ros (danish for feedback) feel free to post them here.

I've also got an open pr which should Hopefully fix most of the async lifetime issues the project has xD

[desert temple]

Fyi i haven't forgotten the open bugs. They will require some rework I've just been incredibly busy the last month. Which should be nearing the end in a week or two and then I Hope to resolve our open issues.

Cheers

[unique pebble]

Rust API with Dagger Go SDK

Throwing out my repo here to show an example on how to deploy your rust API to Google Cloud Run with GitHub Actions and Dagger Go SDK. I was primarily able to follow the docs but I thought it may be helpful to see a repo with more edge cases such as env variables for newcomers.

[desert temple]

I am finally over my busy period so over the next few weeks ill get issues done and build what is required for modules and whatnot.

Priorities:

1. Fix some general errors in using more complex arguments for dagger pipelines this requires a custom graphql input serde serializer.
2. Add examples to sdk to make it easier to get into.
3. Figure if anything special is needed to support modules both in rust but also other pipeline to support rust

[timid karma]

Hey @desert temple, how’s the module support looking?

[frigid ember]

Are there any plans to make Rust work with Dagger Functions?

[frigid ember]

@desert temple Busy again?

[desert temple]

I am not that busy right now, I just haven't gotten into the groove just yet. I personally haven't taken functions into use, but now that it is GA, there certainly is an incentive to do so. I also want the rust sdk to be on par with the others. And honestly it hasn't been for a while, as an example the service support is still quite poor, especially as the dagger team are digging doing more complex function calls.

[frigid ember]

I get back to you if I'm back home

[lunar vale]

Hello! Just FYI there is movement on the #java side, @steady drift and @restive mica discussing best way to bring Functions suppport to Java SDK. Maybe the learnings can benefit Rust as well 🙂

[restive mica]

No pressure 😅

[desert temple]

I've slightly begun to create a custom graphql serializer for rust, such that we can handle complex input objects, as used for services, this should get rust sdk up to par at least with the base features

    #[test]
    fn can_serialize_simple_struct() {
        #[derive(Serialize, Clone, Debug)]
        struct Input {
            something: String,
        }

        let input = Input {
            something: "Something".into(),
        };

        let output = super::to_string(&input).unwrap();

        assert_eq!(r#"{something:"Something"}"#, output.as_str())
    }

graphql input features a slightly different schema from json, as such to support the full spec I've basically got to recreate most of serde_json, that said, I am only gonna implement what we actually use in the rust sdk, so deserializing is not gonna be implemented immediately, which cuts the scope to 1/2, as well as only the actual features required for the sdk

[desert temple]

Dagger now works with the custom serde serializer, as well as all tests passing! It also simplifies the code quite a bit. I am still missing some features, but we're closer than I expected to be able to solve the long standing issue (enum and service issue). Next up solving the graphql errors, and looking at dagger modules

[desert temple]

Okay, I just had a look at setting up modules. It doesn't actually look that bad, as far as I can tell. It is basically just packaging the sdk and codegen code. There needs to be some additions to the sdk loader to be able to load containers not available in the builtin container (shipped with the engine).

The rust codegen is already available as a CLI so that should be quite easy to package, that said I haven't worked too much in the engine itself, so it will require a bit of exploration, and as it seems like rust may be the first sdk to be working on modules, it may require figuring out how Dagger themselves want to include community maintained sdks.

Edit: as well as figuring out how we want the generated code to work. I'll have to experiment but the rust way of doing things is to generate a fake module with all the code, available via. an editor but not in the actual source itself. However, that means that the client has to include the dependencies which I am not sure I dig. So it is probably gonna work the same way as the openapi generator, i.e. the sample project will be a cargo workspace with a main cli crate and an sdk crate. That is likely the proper way of doing things as that means that we can maintain dependencies without interfering with main CLI.

[slim coyote]

@desert temple you wouldn't have had any time to look into https://github.com/dagger/dagger/issues/6933?
i want to propose removing the old ID params from Container/File/etc, but since the rust types are wrong, there's no way to express loading those types from an ID without that

[desert temple]

Sure 🙌

[slim coyote]

thanks thanks ❤️

[desert temple]

A pr is now live for fixing the graphql input errors on some of the more complex dagger options

https://github.com/dagger/dagger/pull/7036

This should now make it possible to port forward containers, hook up services and so on using the rust sdk

[frigid ember]

Tried my best to add some examples to the Rust sdk 😅
https://github.com/dagger/dagger/pull/7097

[desert temple]

I've got a dagger talk lines up for the Rust Aarhus meetup tomorrow but otherwise the plan is to get @slim coyote container id blocking issue fixed. I've got around 4 prs some of them depending on each other it could be nice to get them merged @slim coyote do you want me to just tag you? Or should we do something else. I don't really want to bother you for reviews for what is a community project but i get that you'd still like atleast 2 people involved before a merge. (You as in dagger not Jed)

[dark coyote]

Where is it tomorrow, and hello from a fellow dane in Aarhus

[slim coyote]

Merging Rust PRs

[slim coyote]

Speaking of Rust SDK things - @desert temple how would feel about having the Rust SDK be automatically released as part of our release process? https://github.com/dagger/dagger/blob/main/RELEASING.md
we'd keep the versioning of the crate the same as the dagger engine, and would just do this all automagically

Aside from managing some tokens in our CI, it looks like https://github.com/dagger/dagger/blob/main/ci/sdk_rust.go#L86 is pretty well filled out, so not sure if we have any barriers here? cc @glacial obsidian

[glacial obsidian]

Not from me, wdyt @nimble oxide?

[desert temple]

The crates.io token should already be available, I believe @nimble oxide and I did it initially when rust was upstreamed, if not the dagger team should already have access to the crates.io entry as comaintainers. So it should be quite easy to generate a new token if needed.

Yep, I'd actually talked to @frigid ember about him giving it a crack, it is definitely something we'd like to get done, as I forget to release new versions when we've actually done a few changes.

We'll see whom of us will do it. I've also restructured the rust workspaces such that the dagger-sdk can be released independently of the other ones, which was why we hadn't set automatic releases up initially (because the publish part wouldn't work right).

[desert temple]

The meetup is in MFT Energy in Scandinavian Center. https://www.meetup.com/rust-aarhus/events/299564517/

My talk is Building Rust for Production

[desert temple]

I spent a bit of time getting the last pr ready.

All of it should be good for now @slim coyote let me know if you have anything else still giving trouble for further refactorings/breaking changes

Thanks for your patience

[slim coyote]

Yeah ofc, no worries! Hope your talk went well the other day 🎉

[night quail]

whatchya building in rust lately?

[signal bison]

I'm unsure if I'm struggling with the documentation, but I couldn't find the Rust SDK. Isn't it yet officially released? If not, is there any planned date? (besides Go, I'm planning to develop a couple of modules in Rust ❤️ )

[steady drift]

Hey, it's not official yet and is still in development, you can find the source in https://github.com/dagger/dagger/tree/main/sdk/rust

I'm not sure it also support the module system, you can check the archived doc to find resources on it if you still want to test it: https://archive.docs.dagger.io/0.9/

[desert temple]

@signal bison Rust is currently quite stable with the pre module support.

I haven't dig that much into module support yet, which is why the version of the latest dagger sdk lags a bit behind.

We don't have upstream docs on dagger.io, we do have some examples sprinkled about internally in the source that Tom linked to at least 😄

[steady drift]

Hey, I have an issue with this PR related to Rust: https://github.com/dagger/dagger/pull/7498
I'm introducing the custom Enum definition but it seems enum is a protected keyword in Rust, do you have any idea of a fix @desert temple ? 😄

repro:

• Go to my branch's PR
• dagger call --source=. sdk rust generate -o .

1461 |  pub fn with_enum( &self, r#enum: impl IntoID<TypeDefId>, ) -> Module { let mut query = self.selection.select("withEnum"); query = query.arg_lazy( "enum", Box::new(move || { let enum = enum.clone(); Box::pin(async move { enum.into_id().await.unwrap().quote() }) }), ); Module { proc: self.proc.clone(), selection: query, graphql_client: self.graphql_client.clone(), } }
     |                           ++

error: expected identifier, found keyword `enum`
    --> crates/dagger-sdk/src/gen.rs:1461:177

[slim coyote]

you can prefix the name with r# - see a similar approach here: https://github.com/dagger/dagger/blob/main/sdk/rust/crates/dagger-codegen/src/rust/functions.rs#L17-L23

[steady drift]

Okay I need to figure out now how to say "hey do it for the enum"

OHhh I just need to add the case I guess

Fixed, ty Justin!

[desert temple]

Let me know if you have any other issues 🙏

[sacred trail]

I know nothing about Dagger. Well, actually, I have been following it for quite a while, waiting for the Rust SDK to be "ready" for me to get started with it. However, it has been a really long time. So, how about I help write the SDK myself? Where do I start to understand how Dagger actually works?

As far as I understand, the engine runs a GraphQL server, and the SDKs are just wrapper to help building up the query to run against this GraphQL server, right?

[sacred trail]

Oh actually I see that the rust dev is going on.. just not shown on the website anywere \o/ ... is it usable? missing important features?

[desert temple]

HI Rafael the Rust SDK is usable (I run all of my stuff on it, around 200 builds a day I'd say if not more).

One feature it doesn't have is the new module support. You might see those in the docs everywhere, the dagger sdk right now only supports the pre 0.10 code. I am working on adding that module support in the future, but you should be fine with what is there already 😄

There is a few examples about the dagger sdk in the repository so you can get a feel for working with it:

• github.com/dagger/dagger/tree/main/sdk/rust/ : examples

[slim coyote]

@desert temple rust question for ya 🙂
how does DaggerConn.proc get disposed of? i'm guessing there's something here about dropping, where at the end of main we drop all the references to things, so the underlying process gets SIGKILLed (edit: hm no that's not happening, see "Similar to the behavior to the standard library, and unlike the futures paradigm of dropping-implies-cancellation, a spawned process will, by default, continue to execute even after the Child handle has been dropped.")

a couple questions:

• would it be possible to add an explicit disconnect to DaggerConn? and then leave it invalid to use after, even if there were references remaining
• can we change how the dagger session process is cleaned up? ideally, we should be inline with the other sdks and close stdin, and then wait (so it can tidy up after itself)

i'm not 100% sure how to do these best - i gave it a bit of a go, but i think we'd need to change Query.proc to be a Option<Arc<Mutex<Child>>> (so we'd be able to modify it?)

tl;dr i'm trying to do some sdk tidy-up between how session subprocesses are handled - we hit a weird case recently where if dagger session isn't correctly tidied up, we can end up in weird scenarios where we hang forever

[desert temple]

Rust fun cancellation

[desert temple]

Yet another breaking change 😄

Because dagger didn't properly cleanup connections to the dagger engine, I'd had to change the design to run cleanup routines. As such instead of:

let client = dagger_sdk::connect().await?
// use client

you're gonna have to do:

dagger_sdk::connect(|client| async move {
   // use client

   Ok(())
}).await?;

You can still use the client as you've always done, it just requires this specific scope. The reason is that rust currently doesn't support async drop, and our connection to the dagger-engine requires async.

Rust currently has async drop on nightly, so I may revert this in the future, or at least provide a different function for a more ergonomic handle, once the feature is made stable. But this will do for now.

Other options are requiring multi threaded runtimes always so that I can cleanup dagger in a regular drop, but it will deadlock the program unless tokio multi threaded is chosen (it is the default, just not in tokio::test). As such I've chosen to change the api such that you're required to wrap your dagger stuff in a closure.

[ornate seal]

Hey there. I am testing out this SDK. It is really nice to be able to use Rust in such setting, honestly. Sadly, though, I am getting an error like this using the latest git:

15  : Host.directory(include: [""src", "migrations", "Cargo.lock", "Cargo.toml"], path: "."): Directory!
thread 'main' panicked at /home/deniz/.local/share/cargo/git/checkouts/dagger-b5eab31788856551/aae347a/sdk/rust/crates/dagger-sdk/src/gen.rs:4058:62:
called `Result::unwrap()` on an `Err` value: Query(DomainError { message: "The response is [422]", fields: GraphqlErrorMessages(["Unexpected <Invalid>"]) })
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Any way I can help debug this?

[desert temple]

Hey, are you running this like?

dagger run cargo run

It looks like the version on crates.io has fallen a bit behind the dagger engine version (0.11.9). If you run

cargo run

by itself the dagger sdk will stand up its own engine that matches the version

[ornate seal]

I use the latter. I didn't know I could use it with dagger run.

[desert temple]

Hmm. strange. You wouldn't by any chance have a reproducible example then I could give it a go

[ornate seal]

I have used dagger cli to test other things, though. Maybe it has spun up its own engine using docker. Do you think it interacts with that?

[ornate seal]

Let me see.

[desert temple]

Unless you've set some custom env variables it shouldn't. something like this EXPERIMENTAL_DAGGER<something>

[ornate seal]

No, I didn't.

Would a zip file work for you?

I will swap out my application code with a dummy example, and leave my dagger-related untoched.

[desert temple]

Sure that would work =D.

[ornate seal]

There you go. Hope it helps.

$ unzip test.zip
$ cd test
$ cargo run -p mann server run
  Compiling proc-macro2 v1.0.82
  Compiling unicode-ident v1.0.12
  ...snip...
  Compiling dagger-sdk v0.9.8 (https://github.com/dagger/dagger#aae347a6)
  Compiling mann v0.1.0 (/path/to/test/mann)
   Finished `dev` profile [unoptimized + debuginfo] target(s) in 34.79s
    Running `target/debug/mann server run`
1   : connect
2   :   starting engine
3   :   create
3   :   create DONE [0.8s]
2   :   starting engine DONE [0.8s]
4   :   connecting to engine
4   :   | engine name=c48088b89d6b version=v0.11.9 client=zgcx5ffxgcsprzeybvr1nkfbl
4   :   connecting to engine DONE [0.2s]
5   :   starting session
1   : connect DONE [1.0s]

6   : container: Container!
6   : container DONE [0.0s]

7   : Container.from(address: "cgr.dev/chainguard/postgres"): Container!
thread 'main' panicked at /path/to/cargo/git/checkouts/dagger-b5eab31788856551/aae347a/sdk/rust/crates/dagger-sdk/src/gen.rs:4058:62:
called `Result::unwrap()` on an `Err` value: Query(DomainError { message: "The response is [422]", fields: GraphqlErrorMessages(["Unexpected <Invalid>"]) })
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

[desert temple]

Perfect, I'll have a look.

[desert temple]

Alright. I've replicated the error. I can see you're trying to run against upstream. There is already a crates.io version of the dagger sdk if you want to be less bleeding edge. There are some api differences though 😄

[desert temple]

@ornate seal I found the error. It really isn't pretty, but it is the sed that causes the issues. I'll see if I can handle it for you, it simply doesn't like that there are quotes in the string, and it doesn't properly escape them.

[desert temple]

You can fix it temporarily with:

async fn remove_members(rust: &Container, cargo: File) -> File {
    rust.with_file("Cargo.toml", cargo)
        .with_exec(vec![
            r#"sed", "-i", "s,members = \\[\"mann\"\\],,g", "Cargo.toml"#,
        ])
        .file("Cargo.toml")
}

The reason why it is failing is that it doesn't properly escape the inner " as well as \[ which it also fails on because we just add one extra level of brackets during the parsing to json. (the dagger sdks are basically a fancy graphql dsl client. the r## in rust helps avoid having to escape stuff, sadly our custom graphql serializer isn't so smart, so I'll need to borrow some more stuff from serde_json for handling serialization of strings in a better way)

[ornate seal]

Ah, I see. Well, in that case, I think I'm just gonna move the members section way up, with a fixed line number. And just strip that line instead of querying the file. Come think about this, this is a much better way to handle this. In any case, I think it might be better if I create a self-contained crate for this, instead of knocking the doors of workspace. Seems to bear more problems than it solves for me.

Greatly appreciated nonetheless!

[desert temple]

Sounds good. Yep i do something similar in my own package as well. I'll get it fixed nonetheless it feels unintuitive that what rust accepts the backend can choke on without the user really knowing what is going on

[ornate seal]

Hey, quick question. With the following code snippet, would the target directory somehow be shadowed, even though source directory doesn't contain a directory named target?

.with_directory("target", target)
.with_directory(".", source)

Or, to put it in another way: does it matter which one comes first (or last)?

Also, would target be overridden/shadowed by the contents of source if it were to contain a directory named target?

[lunar vale]

We should really document it, but I think directories are merged with file granularity. So it would not be wholesale shadowing.

By the way this is the same behavior as COPY in Dockerfile (same buildkit operation under the hood)

[ornate seal]

Ah, I see. So, basically you're saying that with_directory (or its equivalent in other SDKs) is COPY in Dockerfile? Then it clarifies the usage for me.

[lunar vale]

Yes exactly

And with_mounted_directory is the equivalent of RUN --mount type=bind,... in Dockerfile. Less exact comparison because Dagger mounts will be re-applied to each subsequent exec, whereas in Dockerfiles you have to re-apply them yourself. But the actual behavior on the filesystem is the same

[ornate seal]

Dagger is a must.

I was playing around with Earthly, and thinking how cool it was. Then read the their FAQ and discovered about Dagger. It is exactly what I always dreamed of. It makes very little sense for programmer tooling to have static configuration files. These stuff need to be... programmed. I really don't like .toml, .json, .yaml files. 😅

[ornate seal]

Would it make sense if Client is non-async at creation, then requires the connection to be acquired later using await:

let c = dagger_sdk::client();
c.connect().await?;

I dont' know much about the workings regarding async drop. This might not be a possible solution either.

[desert temple]

Sadly not as we technically only have a single handle to the sub process (dagger engine). So allocating a new client everytime while possible wouldnt make much sense.

Currently the child process is handled via. Tokio, it could technically be made sync, but would require more infrastructure. Sync drop for async is also possible, it will however require a multithreaded runtime otherwise it will deadlock.

Hopefully async drop pans out that would basically solve all these issues. For now a scoped context seems like our best bet, it is the easiet to succeed with imo.

What are your thoughts on a hard requirement for multi threaded runtime (it is the default for tokio::main, just not for tokio::test)?

[ornate seal]

Just to be clear, you want to depend on a particular library, right?

[desert temple]

Not necessarily. There are some intricasies of how some jobs are spawned in the background that required tokio. I don't know if futures crates has gotten around to spawning tasks yet, tho. I had looked at making it more generic but last time i looked i basically had to write the entire backend of the lib in sync and then write a fancy shell in async to get the same effect ergonomics for the user and as tokio is still pretty much thr standard it just made sense.

[ornate seal]

I see. Well, I don't think I know enough about different async runtimes to provide any valuable insights about this. From what I understand, you are limited by how tokio works and would like to depend on another library that provides better features, right?

[desert temple]

Actually, tokio is limited by what the language can provide. This is also the reason why I've bought into tokio as much as I have for the lib.

Just to give a bit of context:

When async came out, there wasn't really a standard library that was compatible with it, tokio took it on them to provide this, so they have async handles, for file reading, processes. As well as lightweight threads (tokio::spawn). Tokio has been the standard for a while now, there are others such as smol, async-std and futures (sort of). However tokio is like 90% if not more of the async usage.

The async language part of rust hasn't evolved a whole lot and there are still some short comings that are difficult to deal with. Tokio and other libraries have tried to fill that gap. However, there are still some issues, one being async traits. It is currently not possible to define traits for async methods in the same way we do for sync methods. You can use async_trait, but that is a macro (not ideal for a library if we can avoid it). Another is async drop. I.e. being able to call and await on an async drop. It is basically a rust language feature we're waiting for.

If you've ever used sqlx they have kind of the same problem. You may start a transation to the database, and what you get back is basically a drop guard, i.e. it is going to rollback the transaction once the guard goes out of scope. That is the same problem. Sqlx deals with it by doing a sync drop which basically puts a job on queue that is then handled by a background thread. That is fine for that usecase because that is usually a long running process.

As you may've realised a dagger job is usually short lived and the only thing you actually do in the programs associated is just to run the dagger pipeline and then be done with it. I.e. I cannot guarantee that the background job would collect and complete the task like sqlx does.

I may revert the design to be based on variables instead of closures once async drop is stabilized

or another feature comes along, that would do the same thing. I've pretty much tried all the libraries out there, and the only other option I've got for now is to rewrite the core of the library in sync instead of async to avoid using async except for what the user interacts with. That is, making all the api calls and what not blocking to some extend.

That would mean going from

dagger_sdk::connect(|client| async move {
    // Work work
}).await

// to

let client = dagger_sdk::connect().await?;

The latter is actually what the api looks like if you use the currently released version of dagger_sdk on crates.io. The only reason why it changed, again is that I didn't properly shut down the dagger engine, and as such had to change the design to cater to those requirements

Sorry for the wall of text 😄

[ornate seal]

No, no, it was really great. I had grasped your ideas fully.

I used to use the dagger_sdk directly from crates.io. But now I am using it from git directly. And probably will continue to do so for a while. If I can find some time, maybe I can help with things around here (please don't take this as a promise ^^).

[desert temple]

All help is welcome, if you've got any ideas for how to improve the library I am all ears. Especially ergonomics and any surprises if you've had

[ornate seal]

I have been holding off on Rust for more than it was required. Now I am kinda in love with everything (except for compile times).

[desert temple]

I know the errors suck right now, which is definitely high on the to-do list

Rust is an awesome language, it can do so much, it does require a lot of rigor though.

Compile times are manageable once you get the target folder cached properly in ci

My build times for my web services are about 30 seconds or so for a full ci pipeline run (about 10-15k lines of code)

[ornate seal]

Maybe I can help with documentation for now, such as more examples. Regarding the eronomics, I think usages like HostDirectoryOpts could benefit from new()-like initiator methods. Take this example:

.directory_opts(
    ".",
    HostDirectoryOpts {
        exclude: None,
        include: Some(vec![
            "devops",
            "src",
            "migrations",
            "Cargo.lock",
            "Cargo.toml",
        ]),
    },
)

This could follow an API-style like this:

.directory_opts(
    ".",
    HostDirectoryOpts::include(vec![
        "devops",
        "src",
        "migrations",
        "Cargo.lock",
        "Cargo.toml",
    ]),
)

Later, HostDirectoryOpts could take .add_include to append more includes, or add_exclude to create new excludes. Thus, the example could be recrated like this:

.directory_opts(
    ".",
    HostDirectoryOpts::include(vec!["devops"])
    .add_include(vec!["src"])
    .add_include(vec!["migrations"])
    .add_include(vec!["Cargo.lock"])
    .add_include(vec!["Cargo.toml"]),
)

Even add_include could take a single string argument, and add_includes could take the vector one. The same idea could be applied to port forwarding as well.

Or maybe we can apply these to Directory directly. That would get rid of the difference between .directory() and .directory_opts() as well.

.directory(".").include_only(vec!["foo", "bar"])

.directory(".").with_directory() doesn't do the same thing, right? Or rather, can this already be done with with_directory, without_directory etc?

[ornate seal]

Agreed. It just is no nice I have to be exposed to my laptop's heat during these very warm summer days. :)

[ornate seal]

Nitpick: the trait IntoID<T> should really be IntoId<T>.

[ornate seal]

Oh, I just realized there are *Builder patterns. :D

[ornate seal]

For those who might encounter with this in the future: make sure your with_exec statements do not contain newlines or unescaped \, [, " sort of characters. They are not properly escaped when they are sent over GraphQL, even if you do escape them with r#""#.

[desert temple]

I've just gotten of work. I'll probably fix rhe graphql serializer next week which should fix the issue. I'll also fix the naming of intoid I've been writing too much golang lately xD

We should also make the builder pattern more visible. That really help as that is the preferred way to create options

[ornate seal]

Documentation and examples would be the way to go there, I'd say.

I would like to help with that. But I need to get some time off from work, too.

I would really like to help with the Rust SDK. I like Rust, and I have YAML. :)

[frigid ember]

Oh lol didn't noticed that you merged my example PR already 😅
Thank you very much @desert temple!

Is there another task to support you with @desert temple?

[desert temple]

Depends on what you're up for. We still have that IntoID cosmetic change that should be renamed to IntoId instead in the code generator. It is a good first issue to get setup with the code gen. Another slighly more complicated is seeing if we can improve the errors returned. Right now we don't really handle error messages from the dagger engine that well, and just log the output. We should however, be able to unwrap the inner error message and return that directly so that it doesn't look so jank

[frigid ember]

Ok, will take a look 👍

[ornate seal]

Do you think something like thiserror can help with that?

[desert temple]

Somewhat. We're already using thiserror to expose the errors you see. However internally when we call dagger we just get a graphql response back. I.e. just a normal reqwest call, we need to unpack that inner error and return that as an error. Whether that is via. eyre, or thiserror. I can't remember which level it is at there, probably eyre.

In general we expose thiserror errors at the edge, it makes it possible for people to reach to different types of errors. However, for the most part interally eyre is used for convenience.

[ornate seal]

As rather a personal note to myself, but feel free to comment on it: We should change the builder pattern into something like this:

Builder::new()
    .with_foo("bar")
    .try_with_quz("this might fail")?
    .with_bar("blabla")

No finalizers. If it fails, it fails there. If that's not possible, due to it needs a roundtrip to the gql backend, then we can use the _opts function and it fails.

[ornate seal]

Something to be aware of: https://docs.rs/bon/latest/bon/

https://elastio.github.io/bon/blog/how-to-do-named-function-arguments-in-rust

[desert temple]

Interesting may be worth giving it a shot

[ornate seal]

It looks nice, I give you that. But I think in term of ergonomics, what I propose here #rust message is better.

[desert temple]

@ornate seal FYI I've just pushed a pr that fixes the escaping issue we had with exec

Should be out in 0.11.10. I think the engine bump pipeline might be broken. I'll get it fixed an get up on 0.12.2 proper like all the other sdks

[slim coyote]

@glacial obsidian @desert temple just wanted to chat about releasing the rust sdk

i would really like to move rust inline with the rest of the sdks - while that should be automated, i'm not sure about doing a 0.11.10 - there's no corresponding engine release there

can we do 0.12.X?

[desert temple]

@Helder Correia @kjuulh just wanted to

[timid karma]

Does Rust have module / function support yet?

[vapid bolt]

No, not yet!

[timid karma]

The Rust SDK should probably be archived, I don't really see any notable changes within the last 3-4 months.

Unless there's people waiting in the wings to help whip it into shape.

[desert temple]

Rust sdk archive?

[sacred trail]

How will the daggerverse works on the rust SDK?

[vapid bolt]

Hey!

The Rust SDK does not yet support Dagger Modules so there is no way to interact with the daggerverse.

[sacred trail]

Yeah but I'm more thinking if it will ever be supported... How are we going to have it implemented? I think there is no way to add new method dynamically.. Unless we do code generation perhaps a trait that we need to install alongside the module... Or do you have other ideas?

[turbid vale]

From my experience, we need:

1. A module that generating module application and execute the module. Every SDK modules in Dagger use Go as a runtime but can choose the others.
2. The reflection api to reflect the function/method and converting it to module registration. It can be either static code generation like Go does or generate it at runtime (like Python, Elixir, etc.).
3. The execution function that load the function call, invoke the user defined function/method and return it or registering a module.

[sacred trail]

From the user's point of view, they would need to import something... So the newly added functions are added to the scope they are currently in, but where would these functions live? A crate would be auto-published for every module? Would I run a command locally to generate the trait into my source code?

I think unlike JS/Python/Go, Rust is pretty strict with foreign modules; you can only implement traits on structs you own. If these newly generated traits are not part of the dagger-sdk crate, then it would be impossible to have the same DX as what is offered in other languages.

[turbid vale]

where would these functions live?
A new crate (a cli app, maybe).

A crate would be auto-published for every module?
Do you mean crates registry? If it's then it's not necessary. 🙂

Would I run a command locally to generate the trait into my source code?
We use dagger develop to generate the source code into the module. The Dagger will call the runtime to do generate code and put it to the local. (See https://github.com/dagger/dagger/blob/main/sdk/elixir/runtime/main.go#L78 for a reference).

If these newly generated traits are not part of the dagger-sdk crate, then it would be impossible to have the same DX as what is offered in other languages.
On the other SDKs, they vendor the dagger SDK to the module. In Go, it vendor the SDK under internal directory or in Elixir, it vendor under dagger_sdk, and put the generated source code under that directory.

If you're interest in making a Rust module runtime forward, I could help somehow (bootstrap a runtime part, etc.). 🙂

[desert temple]

@sacred trail it should be possible to make the module a lot like Golang, it does share some some rules. You can achieve foreign type methods, using extension traits. I would probably start by going the golang route and see how far I get, and then handle the type rules from there. My initial thoughts was basically just to generate the code, I believe that is what all the other sdks do as well. Both the exposed functions from other modules, as well as the base sdk you are getting via. the module dagger client.

Then use either comments or macros to dilentiate the arguments. Honestly, using macros would be ideal, as it would allow for a typed experience, and should be flexible enough to support the requirements for the module interface, instead of dropping down to comments as some of the other sdks do.

[inner briar]

Hey y'all...hope you're okay
i have this quick question about cloning a git repo using the dagger-sdk for rust
For exmple using the typescript sdk you can do this

    const project = client.git("https://github.com/<GITHUB_USERNAME>/next-app").branch("main").tree()

    const runner = await client.container()
        .from("node:20")
        .withDirectory("/src", project)
        .withWorkdir("/src")
        .withExec(["npm", "install"])

I was hoping I could get some guidance on how to do this in the rust side by mounting the git repo directly into a withDirectory function

[vapid bolt]

Hey!

All of these same methods exist in the rust SDK.

The code would look similar to the examples shown here but instead of passing a local directory you would pass a remote one

        let host_source_dir = client.git("https://github.com/dagger/dagger").branch("main").tree();

https://github.com/dagger/dagger/blob/main/sdk/rust/crates/dagger-sdk/examples/build-the-application/main.rs

[inner briar]

Hello @vapid bolt
Thanks for the insight. I appreciate it.
Let me review my code once again.

[night quail]

how does building a node application in these examples make any sense at all https://github.com/dagger/dagger/tree/main/sdk/rust/crates/dagger-sdk/examples

[inner briar]

Hi guys I am experiencing some funny behaviour with sdk version 0.9.8. I have a bundling service that I want it to receive a message from rabbitmq or thru a REST API call and build a project and pusblish the container.
when i run the function as a script it build okay but it does nothing but print all the info logs...
Any help will be appreciated....Its kind of confusing now after 2wks
The second photo is the same output for when it is called either from rabbimq or the api call

[inner briar]

Hi guys I am experiencing some funny

[runic belfry]

I'm having trouble running dagger init with the Rust SDK.

Since the SDK is experimental, I guess I have to use --sdk=github.com/dagger/dagger/sdk/rust@main rather than just --sdk=rust. Yet, it seems to not be working. I'm getting an error saying "failed to collect local module source deps: failed to get git module sdk: failed to load sdk module github.com/dagger/dagger/sdk/rust@main: select: module name and SDK must be set" despite me setting both --sdk and --name (see my screenshot for full output).

Any ideas??

[slim coyote]

huh that's not a super helpful error
but also, failure is expected - the rust sdk is not currently supported for use in modules - it's only supported as a standalone sdk

[runic belfry]

Oh, alright. I'm new to this, does that mean I can only run it via cargo run, not dagger CLI?

[slim coyote]

yeah, sadly - you'll want to follow the instructions in https://github.com/dagger/dagger/tree/main/sdk/rust

you should also be able to do dagger run cargo run if you want to run the cli as the wrapper

[runic belfry]

I see. Thanks a lot for explaining! 👍

I assume this channel is the best place to follow the development of the Rust SDK, in hopes that it will eventually become a full module with feature parity with Go/Python/etc.?

[slim coyote]

yup, this is the place to be 🙂

[slim coyote]

fyi @desert temple - i've just published v0.15.1 of the SDK on crates.io

https://crates.io/crates/dagger-sdk/0.15.1

published using the scripts defined in https://github.com/dagger/dagger/pull/9197

now that i've determined they work, the pr should now be ready for review!

[desert temple]

Very nice @slim coyote I'll have a look 😄

[slim coyote]

@desert temple first automated crate publish 🎉 https://crates.io/crates/dagger-sdk/0.15.2

now it shall never fall behind again 😄

[desert temple]

You're a hero Jed

[night quail]

this is the coolest thing I have seen all week, thanks Jed!

[modest wedge]

👋 reposting from @pliant verge here: #general message

[pliant verge]

Ah, forgot about the language-specific channels 😅 My question:

Is anyone heavily using Rust and willing to share the functions they are using for building? Looking to optimize the cache heavily to decrease build times.

[night quail]

I am on the lookout for a basic docker build pipeline in rust with dagger

[pliant verge]

So....is anyone building a Rust project with Dagger? 😄

[frigid ember]

@night quail That sounds not that complex could you elaborate a bit more?
I added some examples to the Rust sdk -> https://github.com/dagger/dagger/tree/main/sdk/rust/examples
Not completely sure if they are still perfectly up to date but they should show how to build something with the dagger Rust sdk.

@pliant verge Sorry can't say much regarding 'optimize cache'

[night quail]

just seeking a workflow for building, testing, linting, shipping docker containers for a rust app.

[frigid ember]

Parts of that (building & shipping) are contained in the examples, if you have specific questions, just ask 🙂

[night quail]

I found it sorry for being dumb 😛

[frigid ember]

@night quail nothing to worry about, happy if I could help 🫡

[digital drum]

Which are the roadblocks to make rust sdk officially supported, we would love to know and happy to help...

[frigid ember]

@digital drum Do you mean the module approach from the 3 official SDKs?

[digital drum]

Yes and also in dagger docs

[frigid ember]

I think @desert temple once mentioned he has that nearly working, but that was a few months ago.
He seems to be the only one able to really answer that question.

[digital drum]

ok thanks for the information.

[desert temple]

@digital drum I believe we now have official support for other community based SDKs for modules, so I don't imagine it would be too hard to get support done. I don't have a lot of time right now, and haven't for a while. I'd be more than happy to get someone started on this effort, if not it will be done when I've got the time for it 🙂

@slim coyote have we gotten support for other community sdks for modules? So that we can piggybag of, of their efforts?

nvm: I see that the php one has gotten support which is also community driven.

[slim coyote]

yeah, elixir also has support (and dotnet) - @turbid vale has done a huge amount of work getting those ones working

so those would be the ones to piggy back off of if anyone decides to pick it up!

[desert temple]

I might have a look again now that it is more supported but if someone else want to pick it up id be more than glad to help them get started

[coarse tiger]

Hey 👋

Currently on a trial run with dagger, but I have a bunch of caching related questions, it seems little to nothing is cached locally, and I'd like to understand how/why. For that, I hypothesized that running individual steps with the dagger cli is the right step, but how does the cargo based, dagger_sdk using program map to dagger call arguments?

[modest wedge]

👋 302 to #1342430286331379712 message

[inner briar]

Hey guys? has anyone seen my issue here https://github.com/dagger/dagger/issues/10214

I am having an issue trying to push an image to ttl.sh from the pipeline and I am unable to comprehend what's happening at line 24 to 26 of the logs.
Any hints will help out @desert temple

2025-04-21T10:46:09.064560Z  INFO bundler_service: Server listening on 0.0.0.0:6778
2025-04-21T10:53:22.548089Z  INFO bundler_service::api::bundler: Received request to build 
1   : /home/mt0/.cache/dagger/dagger-0.18.3 session --label dagger.io/sdk.name:rust --label dagger.io/sdk.version:0.18.3
2   : │ connect
3   : │ │ starting engine
3   : │ │ starting engine DONE [0.0s]
4   : │ │ connecting to engine
4   : │ │ [0.5s] | cloud url=https://dagger.cloud/traces/setup
4   : │ │ connecting to engine DONE [0.5s]
5   : │ │ starting session
5   : │ │ starting session DONE [0.4s]
// -----------------OTHER LOGS HERE----------- 

23  : Container.publish(address: "ttl.sh/bun-next-app:2h"): String!
24  : │ remotes.docker.resolver.HTTPRequest
25  : │ HTTP HEAD
25  : │ HTTP HEAD ERROR [1.4s]
24  : │ remotes.docker.resolver.HTTPRequest ERROR [1.4s]
26  : │ remotes.docker.resolver.HTTPRequest
27  : │ HTTP HEAD
27  : │ HTTP HEAD ERROR [1.4s]
26  : │ remotes.docker.resolver.HTTPRequest ERROR [1.4s]

[inner briar]

Guys any information on how I can sort this out...

[turbid vale]

Do you want to submit a PR for it? (Just see this issue for while but have no time to looking at it)

[inner briar]

Hello @turbid vale
I will have to try this before submitting the PR....
I think this may fix the issue: https://github.com/sfackler/rust-openssl/issues/1086

[safe raptor]

#maintainers message are additive options breaking changes for rust SDK users? I had to add new_optional_arg: None in the examples for a recent thing, wouldn't just let me leave it unset

[slim coyote]

wondering if anyone here might have any insight into https://github.com/dagger/dagger/discussions/10479 ? not quite sure how folks are doing x-compilation with rust in dagger

[lunar vale]

Is it a known issue that the Rust SDK does not support Dagger's float type in its codegen? 🤔

https://github.com/dagger/dagger/pull/10680#issuecomment-3105241956

[turbid vale]

@lunar vale it might be a bit late, but the fix is submitted https://github.com/dagger/dagger/pull/10918.

[lunar vale]

amazing thank you!

not too late 🙂

[nova estuary]

I wanted to try and set up a Dagger ci build for my Rust project but I can't seem to be able to make it work.
I tried fuentci and straight dagger setup, but something is broken and it won't work.

If anybody would point me at a working example or even a half-decent documentation guide, I'd be thrilled

[viscid hatch]

Does the Rust SDK support modules yet? I don't see any examples.

[crimson rune]

Anyone using depot as well? I keep getting intermittent failures in github actions do to the cache, wonder if anyone has found a root cause/work around?

[lunar vale]

you mean the dagger cache?

[lunar vale]

<@&1113692274849755156> fans, FYI there is a question about the future of the Rust SDK on github: https://github.com/dagger/dagger/discussions/12070

[proud quarry]

would definitely be nice to have this. i'm guessing it's a bit of a chicken/egg problem. people won't use it until it's mature but it won't mature until people use it and decide to better maintain it. ultimately someone needs to take the risk

[desert temple]

Progress, I've gotten a basic hello world from dagger modules.

@proud quarry I'll move my own companies dagger stuff to modules, once I've got it set up, that should iron out the child pains.

[lunar vale]

Maybe an opportunity to leapfrog the current generation of SDKs, and get right to the new design which other SDKs will gradually transition to.

cc @stray sigil who is working on that 🙂

@silent Maybe we coud do a mini-workshop where core maintainers (led by @stray sigil, @steady drift , @pure pier ) show interested Rust SDK contributors what the ultimate Dagger SDK look like in 2026, and how we're making it easier to achieve

TLDR SDKs getting more powerful, more native, with less effort

[desert temple]

I'd be up for that.

Also

use dagger_module_sdk::prelude::*;

#[derive(Default)]
pub struct RustModule;

/// A simple Rust Dagger module example.
#[dagger_module]
impl RustModule {
    /// Returns a greeting message.
    #[dagger_function]
    fn hello(&self) -> String {
        "Hello from a Dagger Rust module!".to_string()
    }

    /// Returns a personalized greeting.
    #[dagger_function]
    fn greet(&self, name: String) -> String {
        format!("Hello, {}!", name)
    }

    /// Returns the output of running echo in a container.
    #[dagger_function]
    async fn container_echo(&self, msg: String) -> eyre::Result<String> {
        dagger_module_sdk::dag()
            .container()
            .from("alpine:latest")
            .with_exec(vec!["echo", &msg])
            .stdout()
            .await
            .map_err(|e| eyre::eyre!(e))
    }

    /// Builds an alpine container with curl installed.
    #[dagger_function]
    fn build_container(&self) -> dagger_sdk::Container {
        dagger_module_sdk::dag()
            .container()
            .from("alpine:latest")
            .with_exec(vec!["apk", "add", "--no-cache", "curl"])
    }
}

#[tokio::main]
async fn main() -> eyre::Result<()> {
    dagger_module_sdk::run(RustModule).await
}

Sorry to wipe your message of screen @lunar vale

[lunar vale]

One important change is that we're removing codegen from the runtime. So loading a module should never trigger codegen - which makes runtime much faster and simpler

[desert temple]

@lunar vale Do you have a doc / example anywhere I can take a look at for those changes?

The module can now call other modules

[round chasm]

@desert temple i did a few fixes to the rust sdk some time ago for my own use, never took the time to finalize and submit them, I'm not sure if they're still relevant. Let me know if you think it's worth making a pull request: https://github.com/dagger/dagger/compare/main...fdiakh:dagger:enable_rust_tui

[desert temple]

Just looked through the changes, yes I think it could work, I see you fixed the error handling, which I was just about to do, especially unwrapping the graphql strings. I'll leave out some of my fixes until we get yours merged 😉 Feel free to create a pr.

[round chasm]

ok great, i'll do it tomorrow

[desert temple]

Btw, I pretty much have the module support complete, though it does follow the same patterns as the other ones.

[lunar vale]

@desert temple @stray sigil and @steady drift are on French timezone, so when they wake up tomorrow, they can share more 🙂

[desert temple]

I am going to bed now, but here it is: https://github.com/dagger/dagger/pull/12229 I held back on more fixed until we get @round chasm 's pr merged as well so that they don't clash.

Afterwards I'll likely replace some internal dependencies, it takes quite a while to compile for not really a good reason, so I'll see if we can improve that somehow

Edit: I see it has a few failures, will fix tomorrow. Have a good one

[stray sigil]

we'll have that pretty soon 🙂
In short, it's a major change on how SDKs are working. Currently, when we call a module function, codegen is running to generate types from the engine. This codegen is made at runtime, or when you dagger develop. But even if you dagger develop and has generated the files, the runtime will still do it (and sometimes be cached so that's great).
The main change is to now replace dagger develop by dagger generate that is the standard way to generate content (whatever this is on a module or any other code base). And the goal will be to commit the generated files. -> this means the actual code generation will be performed by a normal module, that can be independent from the SDK runtime.
And the consequence is a dagger call to a module will use the already generated files to build/compile the module and run it. No more introspection and codegen at runtime. More predictible, faster cold start, easier to lint, browse the generated code, etc.
This is coming soon for go/typescript/python, currently working on the migration path from the current situation to this one, with some compatibility modules so it's as smooth as possible.

[desert temple]

As soon as you've got a working example / branch do let me know and I'll sniff what you're doing etc. Reducing the runtime cost of building the functions is a nice boon I think 😉

[stray sigil]

This also offers more possibilities. In the end the main change is to decouple the introspection and code generation from the runtime. For instance you could imagine to have a module that will take any kind of input (let's say a yaml describing some stuff to do) and will generate code (in rust) and the runtime will just build it. So you have the rust runtime on one side, but you can have multiple codegen depending on the usage. That can be interesting for compatibility / migration from other systems for instance.

[round chasm]

I've submitted a first set of changes, ill submit the rest once we've dealt with these ones: https://github.com/dagger/dagger/pull/12832

[desert temple]

Review, done. @stray sigil it has been a while since I've contributed, I can see I am still a maintainer, so I don't know if you want to be over it as well 😉

[stray sigil]

Well, I'm not doing rust so it will be hard for me to have an opinion on those changes 😅
So I'd say if that's good for you, if you're comfortable with the changes and CI is green, then I think it's good 👍

[desert temple]

It is no problem, I just want to respect if your processes has changed, if not I'll just steward the changes for now.

Do we have a reviewer for when I need to merge my own changes?

[stray sigil]

Let me check, but either way we'll find a solution. And know we also have the helps of coding agents, for instance to explain me the changes so I can learn enough to validate if needed 🙂
To not hesitate to ping when you have changes you want to merge

[lunar vale]

I think the LLMs are smart enough now to act as a "translator" for a core maintainer to do a useful review even without being familiar with Rust.

For example, to review how the SDK integrates with the engine, whether it follows universal Dagger best practices..

And, as a "smoke test" even for Rust-specific parts. "how idiomatic is this DX for module devs? Are there alternatives? Is this too opinionated? Not enough?"

(I'm not talking about an autonomous review by an AI)

[proud quarry]

with enough tests... maybe it's approach "good enough" for some projects? dagger i think should have human review though, it's well beyond critical threshold