java

[still tundra]

Big news today: there is a POC of a Java SDK 🙂 https://twitter.com/jcsirot/status/1674419646353768449

Jean-Christophe Sirot (@jcsirot)

I published a POC of a @Dagger_io Java SDK! Any comments are welcome
https://t.co/CEczcNnf9Y

Twitter · Jun 29, 2023, 2:08 PM

[still tundra]

https://dagger.io/blog/spring-application

[still tundra]

Linking this help thread which involves a daggerized Maven build: https://discord.com/channels/707636530424053791/1125456542431444994

[onyx mantle]

https://github.com/jcsirot/dagger-java-sdk/tree/build/add-big-jar-with-all-deps#test-without-building

[onyx mantle]

@polar jolt will be presenting an update on the work that is being done on the Java SDK at the the August 10th Dagger Community Call. 🎉

If you haven't already, you can register here: https://dagger-io.zoom.us/webinar/register/9716685521713/WN_USQjVBGXT0SWhNMvqYVvCA

[blissful herald]

Look forward to seeing this!

[onyx mantle]

<@&1113692214514696282> fans! @polar jolt will showcase the Java SDK that he has been working on during the Dagger Community call tomorrow. Don't forget to register here: https://dagger-io.zoom.us/webinar/register/9716685521713/WN_USQjVBGXT0SWhNMvqYVvCA

[blissful herald]

Looking forward to watching this - I'm keen to see this running (possiby) via JBang as well - including the //deps declaration in the script for automated download/install will make calling that even easier.

[still tundra]

Great demo today @polar jolt ! I know @safe bison is very excited 😁

[polar jolt]

thank you

[polar jolt]

I'm looking forward to seeing the PR integrating the SDK merged 🙂

[blissful herald]

rubs eyes at 7am - ewps - missed the call at 4am - look forward to watching the video later on

[hallow lotus]

It's second on my list.

[onyx mantle]

Here is the demo 🙂 https://youtu.be/NgPaxKVaqW0

[polar jolt]

The code showed in the demo is available here: https://github.com/jcsirot/aws-serverless-with-dagger

[polar jolt]

I opened a PR regrading the Java SDK (more precisely the Java SDK build) and I would greatly appreciate a review
https://github.com/dagger/dagger/pull/5826

[blissful herald]

Interesting - you have the maven plugin requiring Java 17? (we've had several discussions on the Maven lists about minimum JDKs, Maven is still 8 minimum). Who/which process actually runs/uses the maven plugin?

I think I should look up some of the examples - hopefully this isn't locked solely to maven?

(looks at the aws example above - ahh, so it does).

ah no - there's no dagger-codegen-maven-plugin there it seems. Where does that get used?

[polar jolt]

To be honest, when I started to work on the Java SDK I did not think about the minimal Java version required for the build. I think this limitation comes from the fact the the dagger-codegen-maven-plugin used during the generated-source phase of the dagger-java-sdk module which is built against Java 17. I don't think it is very complex to make the plugin buildable with Java

But I'm also wondering if it is really required to make the SDK buildable with java 8 since the SDK itself requires Java 17

[blissful herald]

I guess I didn't realise the SDK required 17 either - I've not yet tested it out - I guess that won't matter much once Zenith lands as the build would probably end up using whatever JVM version gets packaged, and won't matter if theres not even a JVM on the host. Sort of - as impressed as I've been with the Zenith examples, practically it (currently) looks like a step backwards - hopefully that becomes clearer in time tho.

It just feels wrong.... having to have java and maven installed, in order to run a build that uses a separate java+maven seems counter. Kinda feels like I'm manging the infrastructure twice. That may not be an issue really tho

[obtuse owl]

Hello, I'm having trouble building the java sdk, I'm getting the following error:

[blissful herald]

@polar jolt this may not be relevant - but thought I'd pass this thread on just in case - https://lists.apache.org/list.html?[e-mail] - I don't think this should matter tho. Still need to check the code, but I don't think you're hitting Sisu/JSR-330 annotations.

[polar jolt]

Are you building the main branch? CacheID has been renamed to CacheVolumeID by PR 5881 and it's probably related

[polar jolt]

Indeed but It will take a look

Hello @hallow lotus Did you find some time to look at this PR? https://github.com/dagger/dagger/pull/5887 Or can I merge?

[obtuse owl]

Yes I'm building the main branch, I pulled the latest changes but I still have the same problem.

[polar jolt]

I will investigate ASAP
Maybe you can try the PR I was talking above

[obtuse owl]

The commit for 5881 has the same error, but the commit right before doesn't

[polar jolt]

which version of the Dagger are you using when building the SDK ?

[obtuse owl]

0.8.8

[polar jolt]

Ok, I understood the cause of the error
The current main branch introduced an breaking API change. The CacheID scalar has been renamed to CacheVolumeID (in order to be coherent with all other ID_able_ scalars where XXXID is a scalar type that represents an object of type XXX). But if you are trying to build a SDK for dagger 0.8.8, the API schema does not contain this change and that's why this error is triggered.
I suggest these workarouds:

• if you really want to build the SDK for Dagger 0.8.8, checout the v0.8.8 git tag of the repo and compile from this tag
• or build a devel version of the CLI and engine (with ./hack/dev), export the _EXPERIMENTAL_DAGGER_CLI_BIN and _EXPERIMENTAL_DAGGER_RUNNER_HOST env vars and then build the SDK

Maybe I could introduce a piece of code which maps CacheID or CacheVolumeID based on the version built but so far I do not know if this effort is worth since the SDK has not been officially released yet

[hallow lotus]

Was in message overload, thank you @calm current for helping out!

[blissful herald]

@polar jolt is the dagger-java-sdk artifact in Maven Central yet at all? It doesn't seem to be at all?

Or do I need to clone and install it locally for now

[polar jolt]

No, it is not yet deployed. I still have to work on the release process automation. For the moment you still have to install it locally 😦

[blissful herald]

Cool - has the domain/account been setup in oss.sonatype.org yet at all? So a manual deployment could be done ( that just requires you to log into nexus and promote the artifacts ), but you can also add the nexus maven plugin to automate that (I really should do that for my own projects, but for now I just promote via the UI as I don't release that often ).

[polar jolt]

Yes, @hallow lotus set the account up (https://issues.sonatype.org/browse/OSSRH-94866)
You can look at https://github.com/dagger/dagger/blob/main/internal/mage/sdk/java.go#L132 to see what I already did to deploy the jars

[hallow lotus]

https://github.com/dagger/dagger/pull/5504#issuecomment-1712123870 + https://github.com/dagger/dagger/blob/22846c00238edaa379b18150540b05174b84d317/internal/mage/sdk/elixir.go#L150-L192 or https://github.com/dagger/dagger/blob/22846c00238edaa379b18150540b05174b84d317/internal/mage/sdk/rust.go#L139-L184 for inspiration.

[glad sleet]

Hello. I try to run the Java samples with the dagger cli (0.9.1) with podman.
The pipeline gets executed but it does not terminate.

./mvnw clean package -Pbigjar,release
dagger run java -cp dagger-java-sdk/target/dagger-java-sdk-1.0.0-SNAPSHOT-jar-with-dependencies.jar:dagger-java-samples/target/dagger-java-samples-1.0.0-SNAPSHOT.jar io.dagger.sample.ListEnvVars

With an explicit System.exit(0); at the end of the main method, the pipeline stops.
Has anyone the same behavior?

[polar jolt]

I never tried podman but I will take a look ASAP

[strange elk]

let us know if you need any help to troubleshoot 🙏

[glad sleet]

We have the same behavior with Mac and Docker.

[polar jolt]

I think I found the bug

[polar jolt]

I opened a PR : https://github.com/dagger/dagger/pull/6121
reviews are welcome 🙏

[slender pebble]

Not sure if anyone else has run into this - I'm able to do dagger run ./gradlew tasks just fine, but running dagger run ./scripts/test.py (which just calls ./gradlew tasks) fails with the error in the image

[polar jolt]

Please, I really need help on reviews on these PR:
https://github.com/dagger/dagger/pull/6032
https://github.com/dagger/dagger/pull/6121

🙏

[safe bison]

We'll take a look @polar jolt . Sorry for the delay!

cc @glossy leaf

@jovial geode

[blissful herald]

This may be relevant to you guys: https://lists.apache.org/thread/9o7tn64nsmcr8n7gyck8xphfysbw16x5

New sisu di update which now supports bytecode >java 14

From memory the last time i mentioned potential bytecode there were no actual issues as the plugin doesn't use sisu.

But still good to know.

[glad sleet]

Hello. There are two Java SDK related PRs to be reviewed, thanks!
https://github.com/dagger/dagger/pull/6319
https://github.com/dagger/dagger/pull/6320

[polar jolt]

@glad sleet https://github.com/dagger/dagger/pull/6320 reviewed and merged

[glad sleet]

Thanks!

[bold matrix]

Anyone building on M1 OSX? My gradle builds are like unbearably slow on my M1

If I point my buildkit host to an external buildkit pod, the build completes way faster.

like 1/6 the time on intel. Running docker 4.26.1 (locally)

[still tundra]

Strange, could it be you're triggering qemu somehow?

Any chance you could share a Dagger Cloud run URL?

[bold matrix]

What does triggering qemu mean? I definitely saw qemu in the runc process somewhere...

Here's my most recent build against the cloud - https://dagger.cloud/FoobarSoftware/runs/eea9a107-6673-4da3-8977-d4be474f60e3 - worked w/in this amount of time when using a remote buildkit pod

Hmm wouldn't let me post in thread...? And hitting a character limit

Anyway here's the process in my OSX docker/dagger daemon. (note qemu as predicted)

/ # ps aux | grep gradle
 1726 root      0:00 /_shim /bin/bash ./gradlew --stacktrace --console=plain -x distZip -x distTar -x ...bunch of gradle args...
 1740 root      0:45 {java} /usr/bin/qemu-x86_64 /usr/bin/java java .... bunch of gradle args.........

Then here's the dagger daemon running on intel/k8s. (no qemu)

➜  ~ k exec -it dagger-86db446cb8-vtrxm /bin/sh
/ # ps aux | grep gradle
 1519 root      0:00 /_shim /bin/bash ./gradlew --stacktrace --console=plain -x distZip ...gradle args.....
 1534 root      0:05 java -Xmx64m -Xms64m -Dorg.gradle.appname=gradlew -classpath /src/gradle/wrapper/gradle-wrapper.jar org.gradle.wrapper.GradleWrapperMain --stacktrace ...gradle args....  -cp /root/.gradle/wrapper/dists/gradle-7.4.1-bin/58kw26xllvsiedyf3nujyarhn/gradle-7.4.1/lib/gradle-launcher-7.4.1.jar org.gradle.launcher.daemon.bootstrap.GradleDaemon 7.4.1

[safe bison]

@polar jolt at some point, it would be amazing to see what we'd need to do to get the Java SDK working on #daggernauts 🙂

[still tundra]

https://tenor.com/view/tamogachilife-zenithentertainment-gif-8001336819991129367

[fringe breach]

Has anyone built a dagger pipeline that auths with aws codeartifact for pulling/pushing artifacts?

[strange elk]

hey! I don't think so but we can help you if you'd ilike to try that out!

[polar jolt]

Hello everyone 👋 I've been AFK here for too long sorry. Actually my job (the on which pays the bills 😅) took me so much time (annual perf review in January…) and also some vacation in February. But from now I should be able to find some time to work on the Java SDK

[polar jolt]

I started to investigate what should be done to "seamlessly" integrate #daggernauts with the Java SDK (Note that if there is any important doc pages do not hesitate to send me some pointers, especially threads from the Discord 😅 I could have missed)

[minor briar]

Welcome back, We are so grateful for all you've done already!

[fringe breach]

Aws and dagger

[still tundra]

@polar jolt @calm current connecting you 🙂

[calm current]

Thanks!!! Let’s work on the module support after the KubeCon 🚀

[glad sleet]

We from Puzzle ITC are highly interested to bring the Java SDK forward. Just contact me to discuss how we can be involved.

[polar jolt]

After the introduction to SDK modules by @calm current during the KubeCon, the first step is make a Java Codegen module. And the first sub-step is moving the codegen part outside of the maven plugin and transforming it into an independant lib (and probably a small wrapping CLI )

[calm current]

I look forward to see your progress! Feel free to ping if you need anything

[glad sleet]

I created a module that generates the schema for any given engine version.
This is an example call to generate the schema for v0.10.2. The generated schema file will be put into the local my-schema folder.

dagger -m github.com/chrira/dagger-module-java-sdk-update/java-sdk@5721074dbd8e7e3470792953775a8a08e78fa24c \
  call \
  generate --version 0.10.2 --dir https://github.com/dagger/dagger \
  directory --path ./target/generated-schema \
  export --path ./my-schema

Eventually we can integrate it to any engine release automation. Or some functions out of it.

[burnt sonnet]

good morning everyone, @glad sleet do you have any news about your java sdk and is it open source ?

[glad sleet]

Hellop @burnt sonnet .
Thanks for asking. We are only doing open source at Puzzle 😉 . The SDK is available in the Dagger github repo. This was added by @polar jolt . He also wrote the java Module.
The SDK has to be upgraded to a Module SDK. This is what is missing.
I'm in contact with Dagger to see how we can achieve that.

[burnt sonnet]

that's fantastic, don't hesitate post here for updates 🙂

do you know if I can already try the sdk ?

also atm the sdk is not in a maven repository we have to build from source I guess ?

[glad sleet]

Yes, you have to build it. See there how to bulid it and run it the old way: https://github.com/dagger/dagger/tree/main/sdk/java

[burnt sonnet]

Ok thanks

[oak granite]

ah there is a Java SDK 🙂

[still tundra]

There is indeed! But it's still experimental, and doesn't support Dagger Functions yet (it doesn't provide a function runtime). So you can use it as a client library to develop custom client code. But you can't wrap that client code in a function, yet

[heavy onyx]

Hi everyone,

I'm working on Daggerizing a Spring Boot application that uses Gradle for its build process. I've written the following Dagger Go functions:

func (m *SpringBootKotlin) Build(ctx context.Context, source *Directory) (string, error) {
    return m.BuildEnv(source).
        WithExec([]string{"ls", "-al"}).
        WithExec([]string{"./gradlew", "build"}).
        Stdout(ctx)
}

func (m *SpringBootKotlin) BuildEnv(source *Directory) *Container {
    return dag.Container().
        From("openjdk:21-jdk-slim").
        WithDirectory("/app", source).     
        WithWorkdir("/app")         
}

The ls -al command confirms that the gradlew file exists within the mounted /app directory inside the container. However, when I execute ./gradlew build, I get the error "gradlew is not such a file or directory."

Any ideas on what could be causing this issue and how to resolve it? Any help would be greatly appreciated!

[grim halo]

gradlew is not such a file or directory

[burnt sonnet]

Hi, does anyone ever tried to compile the java sdk on windows, seems to not working also I found this method running something in sh that's where it fails.

  public static InputStream query(InputStream query, String binPath) {
    ByteArrayOutputStream out = new ByteArrayOutputStream();
    // HACK: for some reason writing to stderr just causes it to hang since
    // we're not reading from stderr, so we redirect it to /dev/null.
    FluentProcess.start("sh", "-c", "$0 query 2>/dev/null", binPath)
        .withTimeout(Duration.of(60, ChronoUnit.SECONDS))
        .inputStream(query)
        .writeToOutputStream(out);
    return new ByteArrayInputStream(out.toByteArray());
  }

I have no idea what could be the workaround for this on windows

@polar jolt do you have any idea how I could fix this ?

[polar jolt]

I'm going to try on my windows box

[burnt sonnet]

thank you so much

[burnt sonnet]

hi sorry but did you got some time to try on windows please ?

[polar jolt]

I was not at home yesterday but I'm going to test it today

[burnt sonnet]

Np take your time 🙂

[burnt sonnet]

I switched to arch linux

[blissful herald]

Are there any precompiled/released versions of the Java SDK yet at all?

[minor briar]

No there are not, I think mainly because its not officially released yet

[blissful heron]

It's in the works tho? Would help me to drive adoption at my company where we've got a split between Go and Java/Kotlin backend services.

[minor briar]

There is no official timeline on Java, but it is definitely something that we do want to support in a first class way.

[burnt sonnet]

it would be great because somehow I can't even compile the java sdk anyway

[open crane]

I tried following the steps in the guide https://docs.dagger.io/integrations/java and on the first try the hung up on the build phase for 15minutes before I stopped it. I then did a manual maven build and noticed that the generated SNAPSHOT version was 3.3.0 and not 3.2.0 as specified in the dagger function. I changed the version in the Dagger function and then it build and pushed the container successfully. This leads to my question: am I correct in thinking the build hung up b/c dagger was not able to find the specified jar file, and if so is there not a fail fast mechanism for this?

[craggy crest]

What is needed to convert the Java SDK to work with modules? We will potentially be interested in using Java to consume our internal modules as 90%+ of internal dev teams are developing Java applications.

[nova elk]

I'm currently working on it 🙂

[craggy crest]

sweet! Is there a way I can follow progress?

[nova elk]

Not yet (I very recently started on this topic), but I'll post there for sure (proposals, PRs) and I'm open to any feedback.
For now I'm based on @polar jolt 's work 👍

[nova elk]

Very early stage, really a PoC, but I have a first version working of Java modules you can try 🎉
Here are the instructions to test:

• use this specific branch of my fork: https://github.com/lgtdio/dagger/tree/javasdk-modularize
• use this specific fork of the hello-dagger example project (the one in the documentation) that already contains a Java module: https://github.com/lgtdio/hello-dagger-java
  From the javasdk-modularize branch:

# enter a dagger dev session
$ hack/dev sh

$ cd where/the/hello-dagger-java/repo/is

$ dagger functions
✔ connect 0.3s
✔ load module 33.0s

Name        Description
build       Build the application container
build-env   Build a ready-to-use development environment
publish     Publish the application container after building and testing it on-the-fly
test        Return the result of running unit tests

$ dagger call build --source=. as-service up --ports=9001:80
✔ connect 0.3s
✔ load module 0.6s
✔ parsing command line arguments 0.1s

✔ helloDaggerJava: HelloDaggerJava! 0.0s
✔ .build(
│ │ source: ✔ ModuleSource.resolveDirectoryFromCaller(path: "."): Directory! 0.0s
│ ): Container! 27.0s

✔ Missing.asService: Service! 5m38s
∅ .up(ports: [{frontend: null, backend: 80, protocol: TCP}]): Void 5m39s

Many things are missing, but that's a first step.

On the things that will come next:

• a nice dagger init to start easily
• a good dagger develop to have an awesome developer experience
• build performances to be improved
• I'll try to shrink as much as possible the dependencies, remove some duplicated, maybe shade some of them. All that to be as agnostic as possible to let everyone use what they want inside the module code

Happy to get any feedback 🙏

[craggy crest]

I want to test this but I am still unable to get a local dev dagger session going 😦 @strange elk I think this is the final piece of the cert puzzle. It seems like the dagger build still won't run on my machine. I get tls: failed to verify certificate errors. Maybe it's doing some dagger-in-dagger stuff where the cert isn't propagated? It's hard to tell. the output is super busy but here's some of it

✘ Container.withExec(args: ["go", "build", "-o", "bin/engine", "-ldflags", "-s -w -X github.com/dagger/dagger/engine.Version=v0.15.2-250116182006-c46f5c2373a5 -X github.com/dagger/dagger/engine.Tag=f61f4cdd3102593f29bf8326cd1c0b3cbe91902a", "./cmd/engine"]): Container! 1.2s
go: downloading github.com/klauspost/compress v1.17.11
go: downloading modernc.org/sqlite v1.34.2
WXgwnDvVZBTP%2F2jleyHPU2Z82FUXyeXDs5vJui18rfVZJlNTa7E2WPO63Q%3D%3D": tls: failed to verify certificate: x509: certificate signed by unknown authority
engine/clientdb/dbs.go:15:2: modernc.org/sqlite@v1.34.2: Get "https://storage.googleapis.com/proxy-golang-org-prod/dcb48d06dd122635-modernc.org:sqlite-v1.34.2.zip?Expires=1737150441&GoogleAccessId=gcs-urlsigner-prod%4
ang-modproxy.iam.gserviceaccount.com&Signature=cSgJa7QNXPYJQTlzOYUuQaJf9eeyPTmv23Mo4VpWcSro%2BLwtIW%2F4Qy1uYz3AwEhIJPRrcHpNQPtOvigwo7Cagv1bz5TzrLCvtQZzJa73gzZmR461VdOQJRWPJdOxX4UMMeOBfBxMacnaayzK9hFPYY3Q%2BwtePeH7wqZE
knKzQfZbWPWLWKIivRM7G5POBr%2Bt7KljFq23PDNFwxeIC0CgO8HK6tU9gnrZol295Xb7lzLCKYFKyx64ns9u%2FeNbeFtr9fvQPS3mDiBaIrV8eoQvudXJiFun324995Zhj1NtOxPFdfBOvDk8Gb7nekNHJxgoGgKIylJjrEtlS2s%2BiA%3D%3D": tls: failed to verify certif
e: x509: certificate signed by unknown authority

[strange elk]

@craggy crest I think I have a very accurate idea about why this might be happening while building the engine. The engine uses a Wolfi container base here https://github.com/dagger/dagger/blob/main/.dagger/build/builder.go#L301 and here https://github.com/dagger/dagger/blob/main/modules/go/main.go#L71-L74 in order to build the engine binary. Thing is, that the cert provisioning logic still doesn't handle wolfi images: https://github.com/dagger/dagger/blob/main/engine/buildkit/cacerts/distros.go#L22-L27

I think you if switch the Wolfi base container and you give it an Alpine one with the same packages and configuration as defined here: https://github.com/dagger/dagger/blob/main/modules/go/main.go#L74-L92 that should work. cc @inner pasture @fresh kestrel

the "correct" fix for this would be to support certificate configuration in wolfi based containers. I think it should be very similar to Alpine since AFAIK Wolfi is pretty much alpine

[fresh kestrel]

🎉 https://github.com/dagger/dagger/pull/9404

probably took you longer to investigate than it took to fix ❤️

[strange elk]

80/20 Jed ❤️. Just approved with some minor nits

[strange elk]

@craggy crest if you pull the latest main and try to build again, that has a better chance to work now

[onyx mantle]

<@&1113692214514696282> fans, check out the news above. All feedback welcome.

Thanks for the contribution @nova elk !

[nova elk]

I just (force) pushed a new version of the java branch https://github.com/lgtdio/dagger/tree/javasdk-modularize
It includes the fixes above.
But the main addition is dagger init --sdk=java is now working, so it might help to test the new java modules 🙂

[polar jolt]

GG @nova elk !

[craggy crest]

didn't work for me 😦 I pulled the latest main. I am not on @nova elk branch. On mainline dagger.

[strange elk]

@craggy crest same error?

[craggy crest]

yessir

[strange elk]

in the same step?

[craggy crest]

I think so, let me confirm

looks like it

 Container.withExec(args: ["go", "build", "-o", "bin/engine", "-ldflags", "-s -w -X github.com/dagger/dagger/engine.Version=v0.15.3-010101000000-dev-bf861d1b99f0 -X github.com/dagger/dagger/engine.Tag=5057d43d674e514ac0d757c22f1728f35c78c2f4", "./cmd/engine"])

[strange elk]

:sus: @craggy crest do you mind modifying the builder code here: https://github.com/dagger/dagger/blob/main/modules/go/main.go#L77 and add the ca-certificates package to see if that fixes it by any chance?

[strange elk]

don't remove any package, just add ca-certificates at the end please 🙏

[craggy crest]

yes! i will try that

[craggy crest]

that did not work, but you know what did work? Switching to alpine. So we are close. Cert is still not propagated to the Wolfi image. This is what I did. Removed Wolfi() and added this.

Container().From("golang:1.23.2-alpine").
WithExec([]string{"apk", "add", "--no-cache", "git", "build-base", "protobuf-dev"}).

[strange elk]

Ok... That's interesting.. since @jed's test specifically checked that the Wolfie image was using the certificate... 🤔

[craggy crest]

right.. There were some caching implications too so i removed my docker volume after testing each time just to make sure the cache is empty. I tested with

1. wolfi, no ca-certificates (did not work)
2. wolfi, with ca-certificates (did not work)
3. alpine (worked)

[strange elk]

@craggy crest ok, I think I know why it doesn't work. It basically comes from here: https://github.com/dagger/dagger/blob/main/modules/alpine/main.go#L138.

We're doing some interesting stuff in our dagger/dagger Wolfi module for caching reasons which ends up using this Alpine module under the hood https://github.com/dagger/dagger/commit/dc18cc39d1ea05bcdc90bc5a8abf1165992797d3. And as you can see from the link here https://github.com/dagger/dagger/commit/dc18cc39d1ea05bcdc90bc5a8abf1165992797d3#diff-aa2bf875ab002944c3f0504299733c497903903ffb80acf15259270ef1a7948bR138, the container that we end up returning from the module is not a "proper" fully blown alpine container with the /etc/os-release file.

I think that if we add the /etc/os-release from the base image in the Alpine module, that should fix it

[craggy crest]

Thanks for looking into it! The last two links you shared don't actually link to the line for some reason 😦 But I think I sort of get what you mean. The change made here is trying to open this file right? How does it work? https://github.com/dagger/dagger/pull/9404/files

[strange elk]

:fake-news: the alpine container by the alpine module effectively contains /etc/os-release

yep.. strange.. the Wolfi container should work. I'll check it out later

[strange elk]

@craggy crest adding the ca-certificates package to the Go module in the builder.go file seems to have made it for me. The way I validated it is by adding ca-certiciates here: https://github.com/dagger/dagger/blob/main/modules/go/main.go#L83 and then calling dagger call --source . env terminal within the ./modules/go directory. That will start a terminal in the same Go env container Dagger uses to build and I could see my certificate correctly configured in the /etc/ssl/certs directory

[craggy crest]

huh.. isn't that what I tried earlier? Let me try that again

[craggy crest]

how are you validating the cert? So I added ca-certificates and also curl (to test). When I do a curl https://companyinternal.url I get a SSL error. I can hit the same url from any other container I run using dagger.

/usr/local/share/ca-certificates/ is empty too. Which is different from what I see in other containers. I see my cert transferred to them at that location.

So there is still a missing piece of the puzzle

[strange elk]

I check that the cert is in /etc/ssl/certs

[strange elk]

This shouldn't be empty

[craggy crest]

That's what I'm seeing.

[strange elk]

Ok... That's definitely strange.. let's check it out on Tuesday 🙏

[polar jolt]

@nova elk The JVM option "-XX:UseSVE=0" does not work with non ARM JVM. I created a draft PR to fix that
https://github.com/lgtdio/dagger/pull/2/file

[nova elk]

Interesting, thanks. I'll have a look. I wanted to remove this flag anyway. I used this flag to run maven on arm64, but it might not be needed in all cases. Especially with orbstack it looks like it's not needed, but it might be with Docker Desktop. I'll check that

[polar jolt]

I'm also working on a couple of changes in the entrypoint java code. But I think we need to sync in order to avoid duplicate work

[nova elk]

I rebased and added more stuff to https://github.com/lgtdio/dagger/tree/javasdk-modularize (and removed the -XX:UseSVE=0 for now)
Especially it's now able to work with dagger develop, exposing the needed generated-sources files to have a good developer experience.
With that I'll see to open a proper PR tomorrow.
I'll focus on having this merged first, then it will be way easier to iterate and add small changes one by one.

[nova elk]

I did it a bit differently: https://github.com/lgtdio/dagger/commit/a710e133b4e5b19d6504875823de93b0158f8568
Instead of adding an entrypoint I'm now setting the MAVEN_OPTS and JAVA_OPTS based on the container's platform. That way, it's transparent for the rest of the code and, for what I tried, works fine with Docker Desktop and Orbstack. And it should work on non arm64.

[polar jolt]

I'm going to test that

[nova elk]

Here is the PR to have Java modules: https://github.com/dagger/dagger/pull/9422

[polar jolt]

I'm going to look at/review the PR

[craggy crest]

Hey @strange elk , did you get a chance to look into this?

[strange elk]

@craggy crest let's check it out together today? I'm not able to repro 🙏

[craggy crest]

sure! lets

[strange elk]

I'll ping you in about 1hr 🙏

[nova elk]

As an example of how the Java modules can work, I'm re-implementing the modules/go module.
Multiple things are missing in the actual implementation (that's the goal of the exercice) and in particular:

• optional/default (what is set in comments in Go, like +optional and +default)
• constructor (to be able to add fields to the object and to execute some code at the object level)
  I quickly drafted what could look like a better module definition, using other annotations for the pragma and record instead of class to deal with the fields and constructor
  https://gist.github.com/eunomie/492c2ca9a0cdf68d0c278db7f766f2d7
  Let me know what you think about it. Especially if there's better / more idiomatic ways to do it.

[nova elk]

Happy to get 👀 /feedback/comments/tests on the java modules PR. That's a first step, as said in the message above multiple things are missing, but that's already a working step.
Especially if that doesn't work on your side 🙂

[polar jolt]

I'm off-site with my company until tomorrow. I'll check this weekend

[polar jolt]

Maybe having a single annotation @Optional(defaultValue = "xxxxx") would be easier to use but since default values ​​of annotation parameters in Java cannot be null there is no difference between no default value and an empty string

Another point: it's a great idea to support records but since the fields of a record are immutable it can perhaps have unexpected side effects

[polar jolt]

I was trying to test java modules dependencies. I rewrote an hello module in java
https://gist.github.com/jcsirot/26f43943bcf7e14055eac9fed07c480a
When the type (here HelloJavaModule) does not have the same name as the module itself, it seems the module is not exposed in the Client interface

Do you know any module exposing multiple types ?

[nova elk]

As the default value should be a valid json string, it might not be an issue. I'll try that way because yes it can be nicer to have only one single @Optional annotation, and if that doesn't work it will be easy to change

[polar jolt]

Ok, after making some tests with github.com/matipan/daggerverse/kubectl which exposes 2 types (KubeCtl and Cli) it seems that what I observed seems to be the expected behavior. Well it means it is not a codegen bug 🙂

But maybe the @Module annotation is not required or it should be put on the module main class rather on the package-info.java file. At least the value parameter of the Module annotation is ignored by the annotation processor, only the description is used to generate the entrypoint.
BTW is there an implicit rule (or explicit but I did not read it anywhere so far) saying the main type of a module and the module itself must have the same name?

[craggy crest]

looks pretty clean!

[nova elk]

Yes, we should have the @Module annotation optional. If that's not already the case, that shouldn't be a big change.

I just pushed some more stuff to https://github.com/dagger/dagger/pull/9422 that includes the support of @Optional and @Optional(defaultValue="\"hello dagger!\"") (or any other default value)
It also adds a test that covers the full code generation, from java sources to entrypoint, this makes it way easier to develop/iterate.

[nova elk]

I thought about this again, and look at the commit you sent me. I did a slightly different change, basically just removing the value of the @Module annotation. We are not using it anyway.
That way, the @Module is optional, and only used to describe the module. But we can define a module with only @Object and @Function.
This is how it looks like to be done for python and typescript, (for instance in typescript @object_type and @function are used)

I pushed this new change

[polar jolt]

Agree, that's an option. However I think we need to enforce that there is at least one class has the same name as the module otherwise no method to access to module is generated in the Client.java (I mean if your module is "foo-bar" then you need to have a FooBar class to generate a public FooBar fooBar(…) method in the Client interface)

[polar jolt]

@nova elk I added a small comment, the Entrypoint.java can generate not compilable code when using primitive types

[polar jolt]

I updated my sample Hello module gist with @Optional annotation and default value. I also added another module using the Hello module as a dependency
https://gist.github.com/jcsirot/26f43943bcf7e14055eac9fed07c480a

[nova elk]

I've fixed the primitive types (for int and booleans) and I'm able to run your example (with one java module calling an other java module)
This works whatever the class name, as soon as either the name in dagger.json is set in the @Object annotation or if name matches the class name itself. And code generation between the modules works great.

[polar jolt]

indeed it is possible to use an alias to match the name

[jovial geode]

@nova elk, just a heads up about the term "optional". There's work still to be done around removing confusion surrounding it, but good to make sure Java starts off clean. Make sure you understand https://github.com/dagger/dagger/issues/6749, if you haven't already. Going through your PR... 🙂

When you vendor a third-party library in a Java application, how is it usually organized? Is there a convention?

[nova elk]

🤔 I see. So basically what we want is to decorelate optional and default value, right? In that case what I've done is not fully compatible. But that should be possible, I'll check that

[jovial geode]

Well, having a default value already makes it optional. Setting something as optional without a default value only makes sense when you're basically saying the default value is null. How does Java handle default values and nulls in function arguments?

[nova elk]

If we set as optional and not define a default value, the argument will be initialized to null (if not a primitive type)

[jovial geode]

Is there a way to avoid having both "optional" and "default value" annotations? People get confused when they have both.

[polar jolt]

I was also thinking about this 🤔 it is something we need to handle. One solution could be to fetch all dependencies and package the module in a fat jar. There are maven plugins which can do that

[jovial geode]

First thing I'd like to know is file organization. Some languages conventionalize on a "vendor" directory, for example. Whatever the common thing is here, I'd suggest using that instead of putting the client library in "sdk".

[nova elk]

There's only a single @Optional annotation on which we can define a field defaultValue. So @Optional or @Optional(defaultValue="my default value")

[jovial geode]

Have you thought about "defaultPath"? Note that I haven't seen the PR fully yet, just getting ahead on some stuff.

[nova elk]

Regarding vendoring, I'm not sure to see exactly. We are not vendoring libraries.
The "sdk" (all the classes, from the sdk/java or generated) are handled differently if it's about the local dev (dagger develop case) or the module runtime.
A first part (the code generation plugin, the annotation processor plugin, the default sdk) is installed in the local (to the container) ~/.m2 directory so that they are available to the user module source. See https://github.com/lgtdio/dagger/blob/bce6fb9e9c2253f669c2ac569463290d605850c3/sdk/java/runtime/main.go#L86-L105
Regarding the dagger develop experience (when we want to generate the sdk code and copy it to the user side so the module can be developped locally) the sdk code + generated code is put under target/generated-sources. This also includes the generated entrypoint. It's not 100% right as some part is not generated, but it might depend how we look at it. See https://github.com/lgtdio/dagger/blob/bce6fb9e9c2253f669c2ac569463290d605850c3/sdk/java/runtime/main.go#L182-L215
And when it's time to run, everything is packaged in one single jar combining the sdk, the generated code, the user code, and all the dependencies.

[jovial geode]

We are not vendoring libraries.

I'd like to understand this a bit more. Can you clarify what's different about what's generated on develop vs call?

All non-Go SDKs essentially copy the SDK's client library into sdk and regenerates the client bindings. This is vendoring. Go does something custom tailored to modules, but still vendors some components, instead of the whole library.

Re: vendoring... ideally what I'd want to do in SDKs would be to use the published library as a dependency and generate the client bindings outside of it, or at least outside of the library's root directory. The problem is it makes testing library changes in modules a bit more difficult. Depending on language of course. There's multiple reasons for wanting to organize this better. There's several different discussions around this (e.g., future module calling on non-module scripts, sdk decoupling, better monorepo support and DX for developing after a git clone).

So I'm basically thinking about future proofing when possible, knowing what's ahead. Changing later is harder.

[nova elk]

Can you clarify what's different about what's generated on develop vs call?
The default way to build java is to have dependencies fetched by maven. And dependencies are copied in the local ~/.m2 directory.
The runtime module is doing exactly that: it takes the java code, generate the SDK code based on the introspection.json provided and package it as jar inside ~/.m2. Then, when it's time to build the user module to create a runnable jar, those dependencies will be used. This is very standard way, and it produces a single fat jar containing everything. The final container representing the module runtime is just a java container running this final jar.

The thing is regarding develop, the local dev experience: as the code needs generation, we can't just have the libraries in a local ~/.m2 directory. It's like in Go, you would like to vendor or to use the dagger.gen.go files. And to achieve that, all the sdk/generated code will be made available under target/generated-sources directory. That way the java build process and java IDEs can find the code, complete, run tests, etc.
In some way that's close to the sdk for the other sdks except it just follows the java way.

I'm not entirely sure to follow to use the published library as a dependency and generate the client bindings outside of it, or at least outside of the library's root directory
You mean to have a dependency to the published library and only vendor the generated code?
Does the published library already contains some generated code?

[jovial geode]

Just the client bindings from the core API, but that's not the point. The general idea (i.e., ideally) would be to avoid the need for dagger develop in the future for IDE integration, after a git clone. If the generated module is a normal (local) java package, where the library is just a normal dependency like any other, and with generated client bindings for the dagger module's dependencies, then you could commit to git those bindings, without having to commit the whole library.

You mean to have a dependency to the published library and only vendor the generated code?

In this case the generated code would not be "vendored". Just generated. Currently, the library is vendored because it's a patched version of the full library, with updated client bindings to include module dependencies, not just core API. I say this in general, with how most SDKs are currently implemented. For example, go doesn't vendor the whole library, just some components.

[nova elk]

I think I see. So to split the library corresponding to a specific engine version, and the extra generated code (for instance to be able to call an other module). And has this part will be small(er), we can imagine to let the user commit it. Right?
But we still need a way to re-generated/refresh it, no? A dagger develop will not be needed anymore after a git clone, but still required when we will reference a new module as a dependency for instance.

[jovial geode]

Yes, correct 👍 Just note that if you install a new module or update a dependency, dagger install and dagger update should update the client bindings as well (i.e., implicit dagger develop).

Of course implementation details aside, the point I'm trying to make is it's better to decouple these if possible because there's different things that would be easier or better if SDKs were organized more decoupled like this.

[craggy crest]

Am I correct in assuming this will use the settings.xml in the user's local ~/.m2? Will the engine pull that in while doing codegen?

[nova elk]

When working on a java module (at least with the current way) it doesn't use the local ~/.m2. The codegen is isolated and only the generated java files are copied locally. The runtime module is using ~/.m2 but not the user one, it's just to split the build between containers. It's not visible from the user perspective.
What behavior you'd like to get by using the user's settings.xml?

[craggy crest]

Any enterprise that uses a private registry for maven will need to use the custom settings.xml. We have to set up our custom repositories and server credentials. settings.xml is the only place you can do that afaik.

[polar jolt]

I agree. I think we may also think about sharing the ~/.m2/repository in order to accelerate the build. Currently maven download all the required dependencies (for the all the plugins for instance) everytime. Or maybe we could use a customized maven image where the commonly required dependencies have already been downloaded

[polar jolt]

@nova elk it seems that one of your recent commit broke the PR

jcsirot@instance-20241122-093153:~/git/my-java-module$ dagger functions
✔ connect 0.3s
✘ load module 2.1s
! failed to serve module: input: moduleSource.resolveFromCaller failed to collect local module source deps: failed to load sdk: failed to load sdk module github.com/dagger/dagger/sdk/java: select: module name and SDK must be set
│ ✔ finding module configuration 0.3s
│ ✘ initializing module 1.8s
│ ! input: moduleSource.resolveFromCaller failed to collect local module source deps: failed to load sdk: failed to load sdk module github.com/dagger/dagger/sdk/java: select: module name and SDK must be set
│ │ ✘ ModuleSource.resolveFromCaller: ModuleSource! 1.8s
│ │ ! failed to collect local module source deps: failed to load sdk: failed to load sdk module github.com/dagger/dagger/sdk/java: select: module name and SDK must be set
│ │ │ ✘ Module.initialize: Module! 0.0s
│ │ │ ! module name and SDK must be set

[nova elk]

I think this is because it's not anymore a builtin SDK. But before the PR to be merged, it's not really available. Once it will be merged, it will be like for PHP or Elixir. In the meantime it's a bit more tricky.
If you want to use the published branch, you can adapt your dagger.json with "sdk": "github.com/lgtdio/dagger/sdk/java@javasdk-modularize"
Or if you want to have it working locally I think it depends where the module and the sdk are. This for instance will work: dad init --sdk=../../sdk/java modules/test-java

[polar jolt]

I keep running some tests and experiments with the java module. I wrote this small weather fetcher module https://github.com/jcsirot/daggerverse/tree/weather-api/weather-api

[jovial geode]

Looking good, it's almost there! Nice work guys! Just some details to cleanup but nothing major.

[polar jolt]

I think we still have to add the support for constructor with parameters in the PR. For instance in this example I have to pass the API key to every function

[craggy crest]

And add support for this? This won't work in my company without it. And I'm assuming others.

not specifically using ~/.m2/repository. That's a bonus. At the minimun, a custom settings.xml should be supported

[nova elk]

I'll have a look, maybe not for the very first preview version, but I'll keep that in mind

[craggy crest]

Sounds good! LMK if you need help testing. I won't be able to use this till that feature is available.

[nova elk]

While adding some stuff to my own todo list regarding the Java SDK, I created an issue: https://github.com/dagger/dagger/issues/9486
That should help to keep track of the progress and have a central place to add needs/requests instead of my own machine 😉

[jovial geode]

Modules support has been merged 🎉

[nova elk]

Thanks 🙂

[polar jolt]

🎉

[polar jolt]

dagger init --sdk=github.com/dagger/dagger/sdk/java my-java-ci is now working

[jovial geode]

In case people don't know, they can try new stuff from a PR with:

dagger init --sdk=github.com/dagger/dagger/sdk/java@refs/pull/9520/head my-module

[craggy crest]

Assuming this won't work on my machine because I need a custom settings.xml. But I'll give it a try

[polar jolt]

I pushed a slighly more complete java module example based on the Dagger Quickstart documentation
https://github.com/jcsirot/dagger-module-java-ci

[polar jolt]

@nova elk I think I found a bug in the annotation processor, the scalar type are not correctly handled by the entrypoint code generation. I'm working on a PR

[polar jolt]

here is a first draft : https://github.com/dagger/dagger/pull/9529/files

[polar jolt]

There is no WithScalar in the Module API. Is it possible to expose a new Scalar from a module?

[nova elk]

What do you mean by not exposed? You should be able to write dag.typeDef().withScalar("Platform") no?
If you have a go module with

func (m *Testgo) Plat(ctx context.Context) (dagger.Platform, error) {
    return dag.DefaultPlatform(ctx)
}

It's exposed as dag.Function("Plat", dag.TypeDef().WithScalar("Platform"))
We should have the same thing in Java

[polar jolt]

while writing my reply trying to explain what I have in mind I think I understood

Actually there is no need to "register" a new scalar, you just have to define use the dag.TypeDef().WithScalar(xxx) wherever it is needed

[polar jolt]

I made a small test. I created a class SemanticVersion extends Scalar<String> and I defined a function returning a SemanticVersion

  @Function
  public SemanticVersion version() {
    return SemanticVersion.from(1, 0, 3);
  }

I patched the annotation processor to generate this :

.withFunction(dag.function("version",
    dag.typeDef().withScalar("SemanticVersion")).withDescription("Returns the version")));

But the dagger functions fails with this error

Error: input: failed to get schema: failed to get schema for module "test-java-module": failed to create function "version": failed to find mod type for function "version" return type: "SemanticVersion!"

actually the PR I submitted is working well as long as the scalar returned by a function or taken as a parameter is not defined inside the module itself

[nova elk]

ok, so maybe we can start with the PR and then continue to enhance. How does that work if you try to do the same thing in Go for instance?

[polar jolt]

I definitely need to investigate. I tried to find a module doing that in the daggerverse but I did not find any so far

I agree, the current PR is a good start

[jovial geode]

Better to find examples in the acceptance tests. When we add features we try to add coverage for them.

[fresh kestrel]

👋 @nova elk just doing release processes for v0.15.4 - the idea is it'll go out today

does it make sense to start:

1. publishing a changelog for java
2. tagging sdk/java/v0.15.4 alongside the other sdks?

actually, scratch that - we probably won't do that for today's v0.15.4, but still stands for v0.16.0 next week

[polar jolt]

@nova elk could you add the sdk/java label to https://github.com/dagger/dagger/pull/9529/ and https://github.com/dagger/dagger/pull/9528/ I do not have to permission to do that
If the PR could be reviewed and merged if everything is ok

[nova elk]

Already merged the scalar one, just waiting for a green CI to merge the second

@polar jolt CI is not happy at all (but I think that's outside of the PR) but if you can review https://github.com/dagger/dagger/pull/9532 that would be nice. This adds @DefaultPath and @Ignore annotations.
I also added more tests to it, but I'm unsure they will pass (it's a bit unclear if they will or not because of the way the sdk is referenced). If they don't pass I'll extract the test commit to a second PR I'll open right after this one is merged.

[jovial geode]

You can commit a module clean from dagger init --sdk=<rel path> in the testdata, then mount that directory in the test as a replacement for dagger init.

Or mount the local sdk in and call dagger init --sdk=<path to mount>.

Can make it work when used directly: https://github.com/dagger/dagger/blob/717788d820fed40deeb8f8476e21627352926bfc/core/integration/testdata/modules/python/extended/dagger.json#L11
Just replace the path if needed, like in: https://github.com/dagger/dagger/blob/717788d820fed40deeb8f8476e21627352926bfc/core/integration/module_python_test.go#L729

This one is more similar to a simple dagger init: https://github.com/dagger/dagger/tree/main/core/integration/testdata/modules/python/pip-lock. Used the same way as what you've already done: https://github.com/dagger/dagger/blob/717788d820fed40deeb8f8476e21627352926bfc/core/integration/module_python_test.go#L1798-L1807

You can replace source code inline instead of a different module source for each test, like we do after running dagger init.

[onyx mantle]

hey <@&1113692214514696282> folks! @nova elk will demo the lastest work on the SDK at the Community Call, so we'd love to see you there. Bring all your questions 🙂

[alpine stratus]

is this for building java apps

or for writing pipelines in java

[still tundra]

this one 🙂

The other was already possible. Now you can build java apps with java pipelines

[alpine stratus]

easy.

go > java so will stick with it 🙂

though imagine the java devs i work with would disagree.

[still tundra]

It's good to have the option!

The goal is that everyone on the dev team can participate in developing and maintaining their build and test environments

[polar jolt]

Too bad, I joined the community call 3 minutes before the end unfortunately 6-7pm is exactly in the middle of my commute

[still tundra]

Sorry JC! That's one of the downsides of being spread out around the world... No perfect time for the call

[nova elk]

I finally found the issue with the tests! That wasn't what I expected 😅

FTR it was issues due to the root when running modules.
Basically I learned the hard way how is defined a root. That is basically the closest top repository that is a git repo, or the closest top repository with a dagger.json.
In my case, there were no git repository and I was trying to reference an SDK outside of the repository containing my dagger.json. So that couldn't work.
And now I understand why integration tests are using a goGitBase even if there's no visible reason to have git here 😅

[nocturne crane]

Does the Java SDK support Kotlin and/or other JVM languages for implementing modules? Maybe not as-is since I guess you'd need to bring an alternative runtime? Asking for a friend who uses Kotlin at work, might make for an easier sell.

[nova elk]

Not at the moment. The focus is currently on the Java version first. But I had in mind Kotlin and clojure as potential other JVM based SDKs. That could be interesting additions.

[nova elk]

Based on discussions the other day, I opened a PR to replace @Nullable by Optional<>. From the tests I've done (and this is covered by some integration tests) it works quite great. @polar jolt @jovial geode
The PR: https://github.com/dagger/dagger/pull/9605
(This is to be merged after https://github.com/dagger/dagger/pull/9604 I split in two to keep them easier to review)

[polar jolt]

I'll try to take a look later today

[still tundra]

@polar jolt @nova elk I would be VERY curious to see what happens if you used the Java SDK to develop a toy AI agent like I describe here: https://x.com/solomonstre/status/1891205257516003344

It's a dev branch of Dagger that adds a primitive in the core API. Should be 100% compatible for all SDKs.

I think this could make things very exciting for the Java community, they're not exactly being treated as first class citizens in the AI engineering community... 😇

[nova elk]

llm works with the Java SDK 🙂

@Object
public class Nono extends AbstractModule {
  @Function
  public Container goProgram(String assignment) {
    ToyWorkspace before = dag.toyWorkspace();
    ToyWorkspace after =
        dag.llm()
            .withToyWorkspace(before)
            .withPromptVar("assignment", assignment)
            .withPromptFile(dag.currentModule().source().file("src/main/resources/prompt.txt"))
            .ToyWorkspace();
    return after.container();
  }
}

I'm looking at creating the equivalent of the ToyWorkspace in Java, but to have it right (at least to duplicate the actual melvin/toy-workspace) I also need to add the constructor support that is not yet shipped.
But that's a start.

[still tundra]

Nice!

[polar jolt]

Oups. Did we break something in a recent PR? I got this error this morning when running dagger functions.

Error: failed to serve module: input: moduleSource.resolveFromCaller failed to get sdk required paths: failed to call sdk module requiredPaths: select: Select: JavaSdk has no such field: "requiredPaths"

dagger.json

{
  "name": "ci",
  "engineVersion": "v0.15.4",
  "sdk": {
    "source": "github.com/dagger/dagger/sdk/java"
  },
  "dependencies": [...],
  "source": "ci"
}

[nova elk]

🤔 That looks weird. I'm able to run dagger functions with no error on my side

[nova elk]

🤔 It has been removed recently. I think one of the issue is between the engineVersion that is v0.15.4 and the sdk source containing new changes.
Can you try with "source": "github.com/dagger/dagger/sdk/java@v0.15.4"? That way it should ensure both versions are aligned.

[polar jolt]

indeed. I'm running the 0.15.4 engine

It works ! Except that @DefaultPath annotation has been merged after the 0.15.4 release and the module does not compile anymore. I think I need to build an engine from the main branch

[polar jolt]

@nova elk I found a commit that works for me 😄

[polar jolt]

FYI: I will give a preview of my Devoxx France talk on how to modularize a CI with dagger modules at the Paris Java User Group on March 11th.

[nova elk]

@polar jolt If you have some capacity I'd love to have some 👀 on https://github.com/dagger/dagger/pull/9605 and https://github.com/dagger/dagger/pull/9523
The second is on top of the first, but in two to keep them smaller and easier to review.
I'm using them on AI agent with no problem, that looks like to work great.
I'll add tomorrow morning some integration tests for the constructor aspect but I don't think this prevent to review them. (<- integration tests are already there to ensure constructor is working)
TIA

[bold matrix]

is the java sdk available from maven central or do I have to build with dagger tooling

[nova elk]

It's not yet available.
But you don't need to build anything, it will be done by dagger itself.
dagger init --sdk=java@v0.15.4 your-module should work (if you have the version v0.15.4 of dagger, this is the minimal version required)
(at some point a dagger init --sdk=java will work)

[bold matrix]

relying on the dagger cli is a non starter unfortunately :/

looking forward to the day I can just add gradle dependency and import dagger though! appreciate the work on this!

[nova elk]

It depends if you want to integrate dagger in your existing app, or if you want to use modules.
For the modules, as there's a code generation aspect, you'll probably have to use dagger cli at some point.
If you want to integrate dagger inside an existing code (without modules), that's different and yes it will be available on maven central. Not yet done, but on the todo list

[polar jolt]

I should be able to free up some time tomorrow

[nova elk]

Thanks 🤗

[nova elk]

One more PR, easy one, to fix float (and other types) support: https://github.com/dagger/dagger/pull/9647

[nova elk]

One more to enable caching for maven (maybe it can be improved more, but that's a start) https://github.com/dagger/dagger/pull/9655

[polar jolt]

I was doing some experiments around support of non Java JVM based langauges, especailly Kotlin and I realized that Kotlin did not support the package-info.java file (where the @Module annotation is put). Even in java actually this file is rarely used, it has 2 purposes: document a java package and put annotations on the package. I was wondering if it would not be wise to put the @Module annotation on the main class of the Module instead. This avoids having to create an additional file that devs are not used to seeing (in addition, nothing prevents a module from being split into several Java packages) and it will unify a little more the DX between the different programming languages ​​on the JVM. WDYT?

[nova elk]

Right now we can run a module without the package-info.java (and so without the @Module annotation). Basically this is only used to get the documentation.
One of the issue with only using the main object is you can have multiple objects in a module. We know which one it is (well, in the sdk-java-constructors branch) so we can handle them differently and put the module description from the main object if we want.
So right now, I don't really know what's best. If I'm right in Go for instance the documentation is also at the object level, not the module/package. I have to check again to be sure.
We can split the module on several java packages, but only 1 @Module is allowed (same, from sdk-java-constructors branch).
That said, I don't have strong opinion, if we have a better way to define the module doc and annotation, or remove the @Module annotation, why not.

[jovial geode]

@nova elk, I didn't notice before that there's a sdk/java/runtime/dagger.json and also sdk/java/dagger.json. Should remove the one in runtime.

[nova elk]

I thought I removed it 🤔 I will add this change

[nova elk]

added to https://github.com/dagger/dagger/pull/9523

[nova elk]

The last changes for the sdk/java introduces an issue when doing a dagger init --sdk=java my-module: all files from sdk/java are copied under dagger-io sub directory of my-module.
Here is a PR to fix it: https://github.com/dagger/dagger/pull/9686

[polar jolt]

I noticed that given how the json serializer was configured in the SDK, it was mandatory that the serializable fields be public. This goes a bit against good practices (and the habits of Java devs) which encourage putting the fields of a class in private.
It is possible to change to json serializer to serialize all fields, including private fields and maybe add a new annotation to mark a field "transient" like in JPA (TBH I'm not sure having such an annotation is necessary)

WDYT?

[nova elk]

yes I know it's a bit against classical practices. In some way I think that makes sense because they are set outside of the class.
We need anyway an annotation, to have "private" serializable fields (// +private in go). My understanding is this is fields that will be serialized but that the user can't set.
We can explore to allow serialization of private fields. And maybe we need two kind of annotations, one for those "~private" fields, one for fields we don't want to be serializable.

FYI the support of constructors in Java has been merged 🎉

And on a sad side, it's currently impossible to read ~/.m2 content from modules (during init, build or execution). I know this is an important ask, and I'm sure at some point it will be possible as it's a shared issue with others SDKs (to access a global configuration for registries/package settings/...).

[polar jolt]

We need anyway an annotation, to have "private" serializable fields (// +private in go). My understanding is this is fields that will be serialized but that the user can't set.
It is also what I understood from the documentation

Actually it is possible to define a PropertyVisibilityStrategy in JsonB in order to serialize/deserialize any field (any private/protected/package and public field) og the module class and we could also introduce a @Private annotation which does not expose the field in the module API but keep the value in order to maintain the module state between calls (which is the same behavior as in the Go SDK)

[polar jolt]

Sorry I did not find much time to review the recent doc PR 😦

Regarding the maven configuration in the pom.xml: it may be possible to create a maven "lifecycle" in order to hide from module developers a large part of this complexity that is currently in the pom.xml

[nova elk]

that's a good idea, I'll have a look at that

If you have some capacity, I have two PRs that can be interesting: https://github.com/dagger/dagger/pull/9707 (to improve code completion on VS Code) and https://github.com/dagger/dagger/pull/9693 to bump and aligned the maven/jre images used

[nova elk]

I've played with the spring petclinic demo to see what we can build in java to build and run java apps.
Here is for instance the Java Dagger function that allows to package, run and expose the application:

  @Function
  public Service service(@DefaultPath("/") @Ignore({".dagger"}) Directory source) {
    return dag.jre()
        .container()
        .withJar(dag.maven().container().withSources(source).pkg().jar())
        .withExposedPort(8080)
        .runAsService();
  }

This is using two small modules I started to build, one for maven, one based on jre to run the java app.

$ dagger call service up

And the app will be available on localhost:8080

You can find the Dagger module under .dagger here https://github.com/eunomie/spring-petclinic
And the two modules:

• maven: https://github.com/eunomie/maven
• jre: https://github.com/eunomie/jre

This is just a PoC, to show what we can do. I'll add more features, use more Dagger features like constructors, etc.
Happy to read any feedback about it 🙂

[craggy crest]

looks good! I did something very similar with Go (for java)

[onyx mantle]

<@&1113692214514696282> fans. help us spread the news! Thank you @nova elk , @visual sundial , @polar jolt , @native junco, @glossy leaf, @safe bison for getting us to this point!

Looking forward to seeing this SDK move to "official" soon.

https://dagger.io/blog/java-sdk

Social posts to share:
-https://x.com/dagger_io/status/1894497196814516453

• https://www.linkedin.com/posts/dagger-io_java-activity-7300263805232521216-ok1E/?utm_source=share&utm_medium=member_desktop&rcm=ACoAAAbjwqUBPZEp__AD2xcvfBAb5pzs_ir11nM
• https://bsky.app/profile/dagger.io/post/3lizsnhuecs2t

Now to find all the Java events that we can submit CFPs too ❤️ 😂

[drowsy bluff]

How does one try it out? Is it bundled with the latest release or ..?

[safe bison]

dagger init --sdk java 🚀

There will be fixes streaming in as we need them, but you can try it now!

[drowsy bluff]

Just to be clear min version of dagger?

[polar jolt]

First talk in 2 weeks at Paris Java User Group. I'm going to rework the abstract with this official announcement

[onyx mantle]

Wouldn't expect anything less of a Dagger Commander ❤️

I know it is a stretch with the short-timeline, but @nova elk was playing with Java agent examples. Maybe the Java User Group would be interested in that too? #agents message

[safe bison]

What version do you need? Prob best results on current version. Even better later this week on next release 🙂

[drowsy bluff]

I prefer best results 😎 Next release it is

[still tundra]

congratulations on launching the experimental SDK!

Here is a question from the Twitter thread: https://x.com/andrenit/status/1894521440503840819?s=46

André “Gru” Fernandes (@andrenit) on X

@solomonstre @marcosnils Does it go well with Quarkus/Graal?

X · Feb 25, 2025, 10:54 PM

[nova elk]

So, I've played a bit with graalvm and quarkus.
(I'm not at all an expert of those, but just to see if that works)
And here is what we can do:

@Function
public Service app(Directory src)
    throws ExecutionException, DaggerQueryException, InterruptedException {
  Directory generatedApp =
      dag.java()
          // inside a GraalVM container
          .graalvm()
          // with the application sources
          .withSources(src)
          // run the install maven command
          // -> this uses by default the mvnw wrapper
          .withMvnExec(List.of("install"))
          // and return the directory with the application built
          .directory("target/quarkus-app");

  Service svc =
      dag.java()
          // inside a JRE container
          .jre()
          // copy the directory we just build
          .withAppDirectory(generatedApp)
          // set the working directory as the application directory
          .withAppWorkdir()
          // expose port 8080 to the host
          .withExposedPort(8080)
          // define the command to be run as a service
          .customRunAsService(List.of("java", "-jar", "quarkus-run.jar"));

  return svc;
}

WIth that, and for instance from the https://github.com/quarkusio/quarkus-quickstarts repository, it's possible to run:
dagger call app --src getting-started up and 💥 the app is running.
dag.java() is a custom module I'm working on, that helps to run maven, jre, graalvm containers

[craggy crest]

I see you moved to a java module as opposed to a separate maven one. I think that makes sense. You can support gradle too later. I am taking inspiration from yours for some of my modules :). I can't use public ones because, company reasons.

[nova elk]

Yep, I tried both approaches. Not sure still which one is best, but I quite like the one with a single module.
That allows me for instance to quickly create one for graalvm: this is just the maven object using a graalvm container and configured to use the mvn wrapper insted of the maven binary (if that makes sense)
At least that's fun to do 🙂

[craggy crest]

yeah, It's also easier to separate the runtime from the build container

[nova elk]

that's one of my goal, to make this distinction easier
when I'm seeing all the docs saying for instance "run mvn install then java -jar ..." it breaks my Docker heart 😅

[polar jolt]

Short question: in the Java it does not seem possible to call the dagger client from a type defined in the module. for instance:

@Object
public class Foo extends AbstractModule {
/* ... */
  public Bar bar;

  @Function 
  public Bar bar(String arg) {
    this.bar = new Bar(arg);
  }
}

@Object
public class Bar {
/* ... */
  public Bar(String arg) {
   /* ... */
  }

  @Function 
  public Container newContainer() {
    return dag.container().from("alpine");  // <- this does not work because it is not possible to inject "dag" into the Bar type.
  }
}

Is it a bug/missing feature/intentional design?

[nova elk]

The way I'm doing it:

@Object
public class Foo extends AbstractModule {
  public Bar bar;

  @Function
  public Bar bar(String arg) {
    this.bar = new Bar(dag, arg);
    return this.bar;
  }
}
  
@Object
public class Bar extends AbstractModule {
  public String arg;

  public Bar() {}
  public Bar(Client dag, String arg) {
    super(dag);
    this.arg = arg;
  }

  @Function
  public Container newContainer() {
    return dag.container().from("alpine");
  }
}

There's no hidden injection of the dagger client, maybe that's something we can add.
As an example this is what I have in the java module I'm playing with: https://github.com/eunomie/java/blob/main/src/main/java/io/dagger/modules/java/Java.java

[craggy crest]

There's no hidden injection of the dagger client, maybe that's something we can add.
Isn't that how the other SDKs do it?

[nova elk]

I haven't yet look at this particular case in other SDKs.
There's automatic injection of the Dagger client for the main object in Java, it's just for the other objects as they are created directly by the user from the code, and not from the dagger java entrypoint.
If an object is created through the entrypoint, then the client is automatically injected

[jovial geode]

The other SDKs use a global var for the dag instance.

Except PHP, which uses a function (e.g., dag()->container()).

[safe bison]

@nova elk could that global var approach work in Java?

[nova elk]

maybe. But if I'm right you can't have a purely global var. It will always be accessed through something else, like a class.
Right now the dag object is an instance of Client. So we could have something like Client.dag() that will return the single object, but I would have something lighter. That's why the client is injected in the class and is available on dag from anywhere in the class.
But this was only for the main object (before I managed to have other objects).
If we can inject the client that would be even better, but if not a global can be a solution.
The main difference is like return dag.container().from(...) vs return Client.dag().container().from(...)
Maybe it's not that important 😉

(Client can be replaced by something better of course, that's just the actual type)

[polar jolt]

AFAIK in the go and typescript SDK le client is almost "static" (from a javaist pov). We could also do the same but it is not really a common practice in Java

Maybe we could automatically inject the Client object in any class tagged with @object where there is a field of type Client

[polar jolt]

The Client field is also serialized ? I don't remember if the Client class implements the IDable interface

[nova elk]

No, the client field is not serialized

[polar jolt]

When module is deserizlzed how the client is set on the other objects than the module main object?

I'm going to look at the entry point generated code

[jovial geode]

The Client object is basically a representation of the Query type in GraphQL (the root type of the API). The only thing "dynamic" about a Client instance is the client connection to the API server. However, especially in a module, you only need one connection, so it's extra boilerplate to have to pass the client instance around. That's why I implemented the global client instance. For more context: https://dagger.io/blog/simpler-reusability-with-the-global-client

[nova elk]

It's not, that's the point. For now you have to provide it in the constructor and call super(dag)
Because that was an easy solution to have it working.
But we can have a look to improve that, or to provide the client in a different way

[polar jolt]

Not sure to really understand how it works 🤔
I will have more time next week to investigate and prepare my talk at the Paris jug

[craggy crest]

I found this to be less ergonomic comparing to how it works in langs like python, TS and Go

[nova elk]

Thinking again about it (while looking at the php demo), and I should be able to do something interesting

• the client object will be available on a singleton like object, maybe under Client.dag() or something else if that makes more sense
• as all module objects are extending AbstractModule (that I should probably rename as AbstractObject...) I can replace the dag field (of type Client) by a Client dag() function that will return the singleton dagger client
  That way, basically the only visible change will be to replace dag by dag() in the user code side. And it will not be required anymore to "inject" the client in any way, and it can be removed from the constructors.
  It will be a breaking change BTW but I think for the best.
  Let me know what you think about it, and if that makes sense I'll see to open a PR tomorrow to change that 🙂

[waxen garden]

Sounds good!

[craggy crest]

That sounds better than the current impl.

[polar jolt]

I like the idea. But if the dagger Client object is statically accessible do we stilll need the AbstractModule class ? You can add this static import import static io.dagger.client.Client.dag in any module object and replace the reference to dag by a function call dag()

[polar jolt]

after looking at the generated Endpoint.java I now understand better how it works

[nova elk]

Thanks for the feedback on the dag client and the different ideas 👍
And here is the PR that moves everything to a single globally accessible client instance with dag(): https://github.com/dagger/dagger/pull/9738

• this is way closer to the other SDKs
• this simplifies a lot when we are using other objects in the same module
• this simplifies json deserialization code (and I think it will makes things easier to deal with private fields, I'll do that soon)
• this means we don't need anymore AbstractModule (it's still there, but deprecated, and will be removed soon)
  -> template has been updated, so a new module should only use the new global dag() client
  All feedback are welcome on the changes, so this can quickly become the new way of writing the module objects 🙏

[nova elk]

❓ for Java lovers:
I'm looking at improving the serialization of fields
The first thing is using private visibility instead of public (I'll not enforce any visibility, just not require public)
The idea is to have all fields serialized by default.
This means if there's a field you don't want to serialize, it will need to be marked as transient. I think that part is OK and really standard for Java.

There's also a different case: a field that we want to serialize, but that shouldn't be exposed to the user.
If you have a field container, by default you will be able to access it using the generated container() method or | container in dagger shell for instance.
But you might want to have an internal field, something part of the serialization but you don't want to expose to the user.

• in go it's // +private
• in typescript if I'm right it's the opposite, exposed fields must have a @field decorator
• in php/python does that exist?

I'll not go with a @Private annotation, and I'd like to avoid a @Field one, I prefer to have as much by default as possible.
So I'm thinking about a @Internal, to say they are handled but not exposed.

WDYT?
(cc @jovial geode as you might have a better understanding of how that works than me 😉 )

[jovial geode]

Go needs // +private because in order to serialize, fields need to be public. If Java can serialize private properties then it's not the same thing. For example, if private fields are serialized by default and you can skip that with transient, then assuming private fields don't get exposed to Dagger there's no need for anything eles.

[nova elk]

like public fields will be serialized and exposed as Dagger Fields
and private fields will be serialized and not exposed as Dagger Fields
Something like that?
I'd say that makes sense to me, but I'm not sure that's the "Java" way to have public fields at all

[jovial geode]

I see. Yeah, in that case it may make more sense to be explicit since you already are with @Object and @Func. But it's up to you guys really 🙂

Whatever's more natural or conventional to Java devs.

[nova elk]

yep, just wanted to be sure to not miss some Dagger behavior here

I have both versions working, so I'll wait advice from java people ( @polar jolt 😉) to see which one is best to pick

[craggy crest]

Although I'm not used to it (using primarily Go), I think @Func to explicitly expose makes sense (Similar to @field in ts)

[nova elk]

Here is also a proposal to have a @Internal annotation to deal with serialized but not exposed fields: https://github.com/lgtdio/dagger/pull/3
It's on my fork for now, just because I'm based on the PR above. I'll refresh that later, but it helps to only see the changes for this feature.
The other addition here is we can serialize fields whatever their visibility, not only public as before.

[craggy crest]

I tried to create a Java module within my corp network. Ran into two issues

1. repo.maven is blocked within our proxy so we need a custom settings.xml (I know this is a known issue but it's un-usable until that's supported)
2. Cert issue. Looks like the CA cert is not available in the Java cert store.

This is from a dagger init --sdk java

I am not sure if 1 makes 2 unnecessary. While pulling from a custom registry will overcome the need for the cert in the certstore, IDK if it will be needed for other things.

[nova elk]

Sadly, currently this is something that is not possible. Modules can't (yet) read files/secrets.
But I wonder if it wouldn't be possible to setup a proxy for that.
It might be less integrated, but maybe a temporary solution.

Regarding the certificate issue, I wonder if it isn't the same issue as 1

[craggy crest]

Well, we can't use the proxy for this. We are not supposed to :). We can only use the internal artifactory registry. This is a common pattern in the Enterprise.

How about allowing the custom engines to add a settings.xml to .m2 folder and using that? Just like how the cacerts are used. The only caveat with maven is you need credentials in the settings.xml to sign in. Right now we take them as env vars

We could maybe use the new engine config Jed created for stuff like creds

I almost feel like Dagger should pull from user's local settings.xml, just like how the engine pulls docker creds from local machine.

Modules can't (yet) read files/secrets.

But the engine can 🙂

[nova elk]

I'll have a new look soon (probably tomorrow, today might be short). Sometimes things are changing quickly and I might have missed some possibilities to fix that.

[nova elk]

📣 FYI I'll introduce some breaking changes
That's why the Java SDK is still experimental.
Those changes will improve the user modules, at the cost of a few modifications to perform. I was thinking about dealing with compatibility, but I prefer to reserve that for later, when the SDK will be in a stable position. Right now it would be an extra cost we probably don't want.
I put a note on each pull request with the instructions to follow if you have a module you would like to upgrade.
-> Please note those PRs are not yet merged, but I wanted to share it before that happen.
Related PRs are:

• https://github.com/dagger/dagger/pull/9767: improving the way we deal with versions (in preparation to publish the libs to maven central, and improvements on caching side)
• https://github.com/dagger/dagger/pull/9738: remove AbstractModule and use a global dag() client

[nova elk]

Just saying, but this Kotlin code works with Dagger 🙂

package com.mycompany.app

import io.dagger.client.Dagger.dag

object Test {
    @JvmStatic
    fun main(args: Array<String>) {
        try {
            println(
                dag()
                    .container()
                    .from("alpine:latest")
                    .withExec(listOf("echo", "Hello, world!"))
                    .stdout()
            )
        } finally {
            dag().close()
        }
    }
}

It's not about modules, but it can be interesting if you want to integrate Dagger in any Kotlin code.
(it can't easily be tested for now, I'm working on publishing the SDK on maven central as soon as possible)

[safe bison]

Kotlin SDK had a very strong request demand when we were polling a lot in the recent past. We should share this! Caveats?

[nova elk]

Caveats? No idea, that's the first time I'm writing some Kotlin code 😅
But it's "just" the Java SDK used from Kotlin code. And for what I saw this looks great, easy to migrate, types where handled as expected

That said, it also means with a small effort/low cost we can imagine to have kotlin modules and not only java... Not my top priority for now, but if it's a strong request this might be worth it.

[minor briar]

Feels like other JVM languages should also mostly "just work™" - ill try with Clojure in a bit just for fun 🙂

[nova elk]

Yeah, I wonder if clojure shouldn't have a dedicated SDK to really benefit from the language. But clojure was on my list of things that could be really nice to see 🙂

[minor briar]

I'm excited to try it out - should i just wait for things to get into maven central or do you think its worth manually wrangling some jars? 👼

[nova elk]

Well... I think that works 😄

[nova elk]

This is not a module (because modules don't support lein for now) but...

(ns daggerclj.core
  (:gen-class))

(defn build-env
  [dag src]
  (let [node-cache (-> dag (.cacheVolume "node"))]
    (-> dag
        .container
        (.from "node:21-slim")
        (.withDirectory "/src" src)
        (.withMountedCache "/root/.npm" node-cache)
        (.withWorkdir "/src")
        (.withExec '("npm" "install")))))

(defn test-app
  [dag src]
  (-> (build-env dag src)
      (.withExec '("npm" "run" "test:unit" "run"))
      .stdout))

(defn build
  [dag src]
  (let [app (-> (build-env dag src)
                (.withExec '("npm" "run" "build"))
                (.directory "./dist"))]
    (-> dag
        .container
        (.from "nginx:1.25-alpine")
        (.withDirectory "/usr/share/nginx/html" app)
        (.withExposedPort (int 80)))))

(defn publish
  [dag src]
  (let [_ (test-app dag src)
        build (build dag src)]
    (-> build
        (.publish (str "ttl.sh/hello-dagger-" (rand-int 10000000))))))

(defn -main
  [fn-name path]
  (if-let [f (resolve (symbol "daggerclj.core" fn-name))]
    (with-open [dag (io.dagger.client.Dagger/connect)]
      (let [src (-> dag .host (.directory path))]
        (println (f dag src))))
    (println "Function not found")))

This is the translation of the Dagger quickstart in Clojure

❯ dagger run lein run publish /path/to/hello-dagger
✔ connect 0.2s

✔ host: Host! 0.0s
✔ .directory(path: "/path/to/hello-dagger"): Directory! 0.0s

✔ container: Container! 0.0s
$ .from(address: "node:21-slim"): Container! 0.4s CACHED
✔ .withDirectory(
│ │ directory: no(digest: "sha256:2e437bda30c530b93370cd4ec12b1a42845812061a22e459bfcefb9023403419"): Missing
│ │ path: "/src"
│ ): Container! 0.0s
✔ .withMountedCache(
│ │ cache: ✔ cacheVolume(key: "node"): CacheVolume! 0.0s
│ │ path: "/root/.npm"
│ ): Container! 0.0s
✔ .withWorkdir(path: "/src"): Container! 0.0s
$ .withExec(args: ["npm", "install"]): Container! 0.0s CACHED
$ .withExec(args: ["npm", "run", "test:unit", "run"]): Container! 0.0s CACHED
✔ .stdout: String! 0.0s

$ Container.withExec(args: ["npm", "run", "build"]): Container! 0.0s CACHED
✔ .directory(path: "./dist"): Directory! 0.0s

$ Container.from(address: "nginx:1.25-alpine"): Container! 0.2s CACHED
✔ .withDirectory(
│ │ directory: ✔ Container.directory(path: "./dist"): Directory! 0.0s
│ │ path: "/usr/share/nginx/html"
│ ): Container! 0.0s
✔ .withExposedPort(port: 80): Container! 0.0s
✔ .publish(address: "ttl.sh/hello-dagger-9429819"): String! 3.1s

ttl.sh/hello-dagger-9429819@sha256:cf38a73be557e801b6381f5894c37598841a5e7a4a197853307d97417a77fbf8

[jovial geode]

@nova elk, left a comment on https://github.com/dagger/dagger/pull/9767. Trying to understand what it changes.

In particular for modules.

[nova elk]

I answered on github, the main change is to set the versions. The final result (what's executed and the different phases to build) is the same. Just it avoids conflicts, improve the cache, and helps for local development as there's not anymore a single version that covers multiple different versions of the lib)

[jovial geode]

I answered on github, the main change is

[minor briar]

This is so cool!

[nova elk]

I finally got an more complex example working. I haven't done clojure for some time, so I'm a bit rusty...
The example is not really important, it's just running its own tests using different versions of the JDK, but it was to have something more interesting

(ns my-app.test
  (:gen-class)
  (:import
    (io.dagger.client Dagger)
    (io.dagger.client ReturnType)
    (io.dagger.client Container$WithExecArguments)))

(defn -main
  []
  (with-open [dag (Dagger/connect)] 
    (let [src (-> dag .host (.directory "."))
          exec-args (-> (Container$WithExecArguments.) (.withExpect ReturnType/ANY))
          exit-codes (atom [])]
      (doseq [v [17 21 23]]
        (let [ctr (-> dag
                      .container
                      (.from (str "clojure:temurin-" v "-lein"))
                      (.withDirectory "/src" src)
                      (.withWorkdir "/src")
                      (.withExec ["lein" "test"] exec-args))
              exit-code (.exitCode ctr)
              stdout (.stdout ctr)
              _ (println stdout)]
          (swap! exit-codes conj exit-code)))
      (when (some #(not= 0 %) @exit-codes)
        (println "Some tests failed")))))

(Maybe there's better way to write it 🫣 )

That's an equivalent to this Kotlin code

package com.mycompany.app

import io.dagger.client.Container
import io.dagger.client.Dagger.dag
import io.dagger.client.Directory
import io.dagger.client.ReturnType
import java.util.concurrent.atomic.AtomicBoolean

object Test {
    @JvmStatic
    fun main(args: Array<String>) {
        try {
            // Get reference to the local project
            val src: Directory = dag().host().directory(".")

            val hasFailure = AtomicBoolean(false)

            listOf(17, 21, 23).forEach { javaVersion ->
                val gradle: Container = dag()
                    .container()
                    .from("gradle:jdk$javaVersion")
                    .withDirectory("/src", src)
                    .withExec(
                        listOf("gradle", "test"),
                        Container.WithExecArguments().withExpect(ReturnType.ANY) // Do not fail on errors
                    )

                val output = gradle.stdout()
                println("Output for JDK $javaVersion:\n$output")

                if (!gradle.exitCode().equals(0)) {
                    hasFailure.set(true)
                }
            }

            if (hasFailure.get()) {
                throw RuntimeException("Some tests failed")
            }
        } finally {
            dag().close()
        }
    }
}

[drowsy bluff]

What we have done with other modules sdks is essentially fork them and then reference a container that has the credentials/certs built in for other languges. It sucks, but once there is a good working example you can tweak it to get it to work. Ideally this should be configurable, but that is a workaround. SDKs are just modules.

[drowsy bluff]

So in this case we would change this: https://github.com/dagger/dagger/blob/main/sdk/java/runtime/main.go#L354 to reference an image that has creds and CAs. And then rehost the sdk module.

[craggy crest]

I see. Good to know that's an option although at this moment I don't want to maintain my own SDK. For using a custom SDK you give the full git path to dagger init --sdk?

[nova elk]

🤔 But can you reference this image during a dagger init for instance?
To be able to override the image doesn't really looks like a good idea, you have to stay in sync with the evolutions of the sdk for instance (versions, tools, etc)

[drowsy bluff]

Yep. Just the git module address.

[drowsy bluff]

Other sdks allow you to override the image via config.

https://github.com/dagger/dagger/blob/main/sdk/python/runtime/main.go#L45

[next sundial]

Is anyone online

Hello

[polar jolt]

@nova elk I maybe found a bug in the Java SDK while working on my today's talk. When using InputValue objects (actually I wanted to use the PortForward class) the Entrypoint.java class does not compile. Unfortunately I hadn't time to investigate further but I will try to understand what's going on and generate a minimal test case.

Caveats? No idea, that's the first time

[nova elk]

Let me know if you want I have a look, if you have for instance a small test case I can try

📣 Small change on the Java SDK:

• all fields, whatever their visibility (including private) can now be serialized
• if you have a field that shouldn't be, mark it as transient
• all public fields will be exposed through a Dagger function (with the name of the field). For other visibilities, add @Function annotation to the field
  That way it's closer to other SDKs like Typescript and easier to use.

[polar jolt]

Let me know if you want I have a look,

[nova elk]

🎉 I finally have enum support for the Java SDK. PR is here if you are interested: https://github.com/dagger/dagger/pull/9856
The only downside of this solution is it requires to add a @Enum annotation to the enums defined in the user module code. I can't get it to work without, because the enum doesn't exist at the time we are parsing the annotations to generate the entrypoint.
But that's a small downside, otherwise this works as expected 🙂

[burnt sonnet]

Hi, is it possible to use the dagger sdk with gradle kts ?

I followed the dagger documentation, I have init and setup the java sdk with dagger functions as recommended but it seems I don't have access to the sdk in my code

[nova elk]

If you built the jar locally (as they are not yet published) then you should be able to call it from any jvm language. I have it working with Kotlin and clojure in addition to java (maven and gradle).
You can find some more information on this preview of the docs: https://deploy-preview-9798--devel-docs-dagger-io.netlify.app/api/sdk It's a preview because it needs the jars to be published but this should work with local built jars

[nova elk]

In this case (with custom apps) there's no access to modules.
Kotlin support to modules is something I'm thinking to add at some point, but it will come a bit later

[burnt sonnet]

do I still have to use dagger init command ?

i'm just not running the develop command ?

[nova elk]

no, dagger init/develop is about modules, and not supported with kotlin. It only supports java - maven at this time

[burnt sonnet]

ok

[burnt sonnet]

I think it would be good if the cli would ask what build tool we use

like it could detect our build file and add the dependency

[nova elk]

That's something that will come, yes. But it's more than just asking/detecting which one to use, they have to be supported as in case of a module the build is done by the module runtime, not directly on the host side.

[burnt sonnet]

do you know if the java sdk supports long running services ?

[nova elk]

The main idea is the Java SDK should support what Dagger supports, the different SDKs are quite small in fact, letting the engine do the hard work. (that said we can have missing pieces as the Java SDK is still in progress)
So if by service you mean services exposed through a Dagger Service object, yes they are working: https://docs.dagger.io/api/services

for instance you can have a module that will start a mysql service and make it available to others or to the host

[burnt sonnet]

Thank you

[nova elk]

no problem. Happy to get any feedback about how that works, how it can be better, what doesn't work, etc 🙏

[craggy crest]

Hey @nova elk , now that this is merged https://github.com/dagger/dagger/pull/9323 is it possible to use that to set default path to a custom settings.xml?

[nova elk]

Yes, I'll have a look to see how the Java SDK can use that 👍

the sdk.config could also be a place where we can set info like maven/gradle and java/kotlin/clojure in the future

[fresh kestrel]

@nova elk any idea why we're seeing failures on main related to java?

https://v3.dagger.cloud/dagger/traces/89f0a05a5f4b561b929fe182aa474f62?listen=7498c82ffd53b3fe&listen=0be3c5c4bd939562&listen=48f159bf4055f026&listen=b26a13fa11633040#b26a13fa11633040:L382

27 : [16.4s] | [ERROR] Failed to execute goal org.apache.maven.plugins:maven-install-plugin:3.1.2:install (default-install) on project dagger-java-sdk: Failed to install metadata io.dagger:dagger-java-sdk/maven-metadata.xml: Could not parse metadata /root/.m2/repository/io/dagger/dagger-java-sdk/maven-metadata-local.xml: in epilog non whitespace content is not allowed but got > (position: END_TAG seen ...</metadata>\n>... @13:2) -> [Help 1]

just seemed to have started on the most recent commit? which is confusing, since that's a php sdk change lol

[nova elk]

Could it be related to the other failure (testdev-cgroupsv2): tee: stderr.txt: No space left on device? I don't know how the CI is setup, but if we don't have space left we can have weird errors I guess.
/root/.m2 is a cache volume, is there a way we can prune it?

[fresh kestrel]

those failures should be fixed by https://github.com/dagger/dagger/pull/9887, I think they should be unrelated...

[nova elk]

I don't really see reasons why the test would fail that way. I mean that's not the test that is failing, it's the build of the SDK, on one test. So maybe something's wrong has been written, but not sure why (is there a way we can inspect a cache volume?)
The test is passing on my machine, and hasn't changed.

[burnt sonnet]

I will give you a feedback tmr, I have not yet implement it

If you are interested why I need dagger is because we use a framework to develop our plugins on our Minecraft server and this framework has a gradle plugin that start a test Minecraft server but its not using docker yet so i'm migrating it to use dagger

[nova elk]

If anyone has some capacity ( @polar jolt 👼 ) I'd love some 👀 on https://github.com/dagger/dagger/pull/9856 Basically I rewrote the way we are dealing with types to fix multiple issues regarding them and especially List (more details in the PR body)

[nova elk]

Custom maven settings.xml

[fresh kestrel]

cc @nocturne crane

[polar jolt]

I'm trying to find some time to review that PR. Sorry these past few weeks have been busy at work 😦

[short rapids]

Hi, Just testing a complete clean startup experience with the Java-SDK (many thanks for the progress there ♥️ ) and don't have anything in my local maven-repository.

• After a "dagger init --sdk=java"
• i import the folder in IntelliJ: "idea ."
• a maven refresh can't find the annotation-processor dependency.
• i then built the annotation-processor dep manually and installed it locally. then changed the property in the pom (see images)

from my understanding the annotation processor should also be part of the generated sources by the "dagger develop" command so that i don't need to mess with external dependencies to get started. Or should some dependencies (dependency versions) be decoupled from the engine releases and be distributed externally via jCenter / Maven Central in the future? I think the "getting started experience" should be free of any obstacles. Already any plans for this?

[nova elk]

First, thanks for trying 🙂

don't have anything in my local maven-repository
That's fully expected.
In fact your module will never be built or run on your machine directly, it will always be through a Dagger module runtime.
So the dependencies are built there (inside the runtime) and are never needed locally. In fact you can't build locally (maybe that will be something in the future, but not now)

The target/generated-sources folder should contain the needed files that helps your IDE to have completion for instance.

The JAR libraries are not (yet) published. They will be at some point.

[short rapids]

Thanks I was expecting that. What i‘m trying is to find out how to quickly add an run unit tests for a module in a „Java native“ way out of my ide. Any ideas, or is it just: make a separate module as a test harness for that and use dagger to run Java to run dagger to run Java? I would be ok with that. Just trying to find out which direction to explore

[nova elk]

That's a good point. Currently, there's no magic way to do that. The real solution will be to publish the libraries (I hope we get there soon) and generate the code for the module (that's a bit different from the libs) in a way we can have all the deps locally

[craggy crest]

Hey @nova elk , any updates/thoughts around how you can support custom settings.xml? We have a massive Java presence and they will not adopt dagger without being able to build and run modules with Java :).

[nova elk]

No yet 😦
This is still on the top of the list, I'll have a new pass on that very soon.
You might also be interesting by this issue: https://github.com/dagger/dagger/issues/9991
This doesn't cover the exact same thing (especially if you add an extra lib) but for the basic work it means you should be able to init and use a module without any access to maven central. That's the goal.

[craggy crest]

Thank you! Yes, I'm closely watching that too.

[magic rover]

Good morning,
I'm trying to create some pipeline with dagger with java SDK but I have some difficulty to find the javadoc and the ide not resolve the imports,
Could you help me?

[nova elk]

👋
Are your trying to use it with modules (based on dagger init --sdk=java) or by using more a "custom application" approach like https://docs.dagger.io/api/sdk#custom-applications ?
The java doc is not (yet) published as well as the sdk jars. But if you are using modules, your IDE should have access to the code and help you with code completion for instance.

[magic rover]

Good morning @nova elk I'm using dagger init command and sometimes are resolved and sometimes no

[nova elk]

The way it currently work is the dependencies are defined and build inside the Dagger java runtime. And because they might depend on the module (especially if there are dependencies) they are not exposed on the host. But the files are all exposed under target/generated-sources. This is just a convenience for local development, they are not used to build, but allow to have code completion.
So basically the dependencies in the pom will not be resolved on the host, but only at build time inside the runtime.
That's something that will change at some point as we are working on improving all this area.
Do you have code completion working still? If not, what IDE are you using?

[magic rover]

I saw them but I don't understand deeply how does it works and when they are generated... Actually I'm using vscode
Is very difficult to apply some change without class documentation and completion
Ps. The same "issue" appear with go and typescript

[nova elk]

🤔
You should have access to code completion and docs in all cases, java, go, typescript.
See for instance this recording I just made, I create a new module, open vs code and got instant code completion and docs.
Do you have the corresponding files under target/generated-sources?

Code is generated at runtime (so when you use dagger call or the dagger shell to call your module).
The code is also generated when you dagger develop. This command will create the needed files on your host but they are not used by the runtime, they are here only to help with IDEs.

[magic rover]

Okok many many thanks for your explanation... I will try and report you a feedback

[nova elk]

No problem, do not hesitate to ask for help 🙂 I'm more than happy to get feedback on the actual state, and fix what needs to be.

[magic rover]

I have another question there is a way to run dagger engine inside a docker compose instead of an invidiual container?
My scenario is: Im trying to create an complete environment to develop and test a ci cd at the moment I have Jenkins, gitea and nexus but when I run dagger module from local git or push to local nexus dagger cannot see them
How can reach the goal?

[nova elk]

Not sure to see exactly.
Regarding Jenkins you might be interested by https://docs.dagger.io/ci/integrations/jenkins/
I'm not sure if that can run on compose, but the idea is the dagger cli will start the engine, so you don't have to provision it by yourself.
When you say "when I run dagger module from local git" what does that mean exactly? Is your module on a local git server? And how are you trying to access it? My guess is a dagger -m your-git-url should work (just to display available functions). You can have a look at https://docs.dagger.io/api/remote-modules that explains the different auth methods

[magic rover]

Yes i follow both guid they explain to me and they very clear
Sorry I explained very bad, try to clarify
I have a docker compose with
Jenkins
Gitea
Nexus

Gitea contains 2 repo one for dagger modules and one for application

When I start the pipeline of apllication repo, dagger not found gitea or nexus

Ps. As a workaround I cloned the dagger modules repo before dagger call command and use module from file system. In this case nexus cannot be resolved by dagger docker engine.

[nova elk]

ok, I think I see. The issue here is a docker compose will create its own (docker) network. And the auto-provisionned dagger engine will not be aware of this network.
So you need to run the engine manually and connect it to the right network to have access to your components.
Based on https://docs.dagger.io/configuration/engine/#configuration and https://docs.dagger.io/configuration/custom-runner/ you should be able to either:

• run an engine on your own and connect to the right network
• integrate it in your docker compose stack
  And you'll probably need to export _EXPERIMENTAL_DAGGER_RUNNER_HOST to allow the dagger client to connect to its host instead of trying to start one

[magic rover]

hi @nova elk , many thanks for your support about code completion there is an issue on my ide now is ok... about the dagger runner i successfully configured on docker compose and connect to jenkins but i have some issue with pushing artifact/images to nexus but it seems is a configuration issue of ssl... is very annoying

[magic rover]

this is my actual repo https://github.com/robertonav20/dagger-modules maybe could be interesting or useful to the others

[magic rover]

hi @nova elk,
i made others test after successfully configuration of nexus as docker registry with nginx.
From my wsl i can login and push images to the repo but on jenkins no

could be depends on docker, because is mounted from host instead of runner

[nova elk]

this looks like a dns/network issue. From jenkins, is nexus.local available? For instance if you open a shell on your jenkins container. Did you try with just nexus and not nexus.local? I might be wrong but that's my understanding from https://docs.docker.com/compose/how-tos/networking/

[magic rover]

hi @nova elk ,
you are right there was network issue fixed with "links" field of docker compose and how i extract and pass credentials to dagger function. now is working! many thanks
Anyway i lose a lot of time to configure nexus as multi artifact repo behind nginx, it was very hard found the correct guide

[nova elk]

Really happy that works 🙂

[nova elk]

Small Java addition, based on the recent config map available in dagger.json, the ability to enable -e and -X on maven commands: https://github.com/dagger/dagger/pull/10247
This should help understanding when something goes wrong (for instance regarding maven dependencies)

[craggy crest]

Didn't know that was possible. Something similar could be used to point to a local settings.xml right? command is mvn -s settings.xml. Question is, how does the module pull it in

[nova elk]

no, it's still not possible that way

the module runtime only has access to the sdk source dir and to the module source dir
🤔 Well, while writing that I wonder if we couldn't have a specific settings.xml there. Maybe.
But that means dagger init --sdk=java will not work, you have to do that in multiple steps
But maybe

But it can't be your settings.xml you have in your ~/.m2

And there's also the question of auth info that could be outside of the file

[craggy crest]

I would be fine with that. If I can upload a custom settings.xml in the module repo AND set the credential when doing dagger init --sdk=java --user env:USER --pass env:PASS. At least that's basically what we need. There could be other creds in their so, it's not all encompassing.

Dagger already pulls in docker and git credentials from local. I'm afraid java SDK will have to do something similar with .m2

[still tundra]

Dagger already pulls in docker and git credentials from local. I'm afraid java SDK will have to do something similar with .m2

This would be worth an issue...

[craggy crest]

Maybe a way to pass the creds into the container via dagger.json?

{
  "name": "foo",
  "engineVersion": "v0.18.4",
  "sdk": {
    "source": "java",
    "config": {
      "mavenErrors": false,
      "mavenDebugLogging": false
      "credentials" : {
         "USERNAME": "env:USER"
       }
    }
  }
}

[still tundra]

Then that module will only be usable by one user..

[craggy crest]

oh? I was thinking that config would be on the consumer side. Not the producer.

But now I get it. Even the consuming side module will be tied to a single user. I'm stuck in module-ception 😄

[still tundra]

https://tenor.com/view/denaziamigo-doggo-dog-puppy-weird-gif-17908738

[nova elk]

Yes, that the thing, dagger.json is not user specific. But the settings.xml is. So if we want a module to be re-usable it should not contain anything user specific.
I started to look at adding it to the engine, as it might be possible to load it, like docker/git creds. But it's not yet finished.

[still tundra]

Is settings.xml a user-wide configuration file similar to ~/.ssh/config?

[nova elk]

Yes, this is the default configuration file for maven: https://maven.apache.org/settings.html

[craggy crest]

Yes, and it allows usage of env vars. For example, you can specify env.MY_TOKEN inside the settings.xml and mvn will substitute that at runtime with the MY_TOKEN env var. This is nice because settings.xml can technically be checked into the repository (we do in places).

[magic rover]

Hi,
i'm trying to publish a jar to nexus repository from jenkins with a dagger module. When i execute the step Deploying Artifact after some seconds i got this issue "Error: input: javaBuild.publish process "mvn -X deploy:deploy-file -DgeneratePom=true -DrepositoryId=server -Durl=http://nexus:8081/repository/maven-releases -Dfile=app.jar -DgroupId=com.example -DartifactId=hello-world -Dversion=1.0.0 -Dpackaging=jar" did not complete successfully: exit code: 1" but i don't understand why.
I've checked the logs of dagger engine but i didn't find relevant logs to understand the reason
The screenshoots contains jenkins logs of pipeline and snippet of step executed
Could you help me?
Thanks in advance

PS: i've tried the same command locally on cli and it works correctly

T

[safe bison]

Hi @magic rover 👋 Could you provide a DAGGER_CLOUD_TOKEN on your Jenkins executor so you can provide a Dagger Cloud URL for us on the the Dagger team to look at?

Also, is there an easy-ish for us to reproduce?

[magic rover]

Hi @safe bison,
I'm sorry but I didn't bind to dagger cloud, if you want I can share the GitHub repository https://github.com/robertonav20/dagger-modules

[nova elk]

It's hard to know exactly what's going wrong, but based on the code you only display stdout. You probably also want to read stderr, the error might be visible there.
One other thing you could to is to insert a terminal() instruction just before the withExec(mvn deploy:deploy-file) so that it will gives you a terminal inside the container, with all the env variabled and files. From there you can run the command manually and see what's wrong and then adapt the command accordingly.
The same way if you add a -i to the dagger call command it should automatically spawn a terminal on exec commands.
All that don't solve directly your issue but that should help to understand it.

[magic rover]

Hi @nova elk
understood I will try as you suggest maybe just with stderr will show me what's happening.
Many thanks
But I'm expecting to se all logs of each step executed maybe i miss something?

[nova elk]

Did you tried with more verbosity? -v to -vvv?
And dagger cloud might also help to dig into this kind of issue by giving a better way to navigate the traces

[magic rover]

hi @nova elk
i follow your suggestions with "terminal" there is a communication problem between maven container and nexus, they are inside the same network of docker compose
PS. stderr() not works as expected i didn't see logs from jenkins just this

[magic rover]

@nova elk many thanks i fixed it!
Prerequisities _EXPERIMENTAL_DAGGER_RUNNER_HOST="tcp://localhost:1234" this env var able me to connect to container used by jenkins and reproduces the issue

1. put "terminal()" before of command execution
2. with manually execution of command i understood the issue
3. fix nginx configuration
4. rerun with correct address

[nova elk]

Happy it's working 🙂

[magic rover]

@nova elk could be possible to see logs of commands executed during pipeline? Is possible only with use of dagger cloud?

[nova elk]

I'd say most of the time we don't see them directly (I mean it depends how you handle your command and stdout/stderr)
But you can handle that in your code: https://docs.dagger.io/cookbook#terminate-gracefully
Translated in java it's something like

  @Function
  public String test() {
    try {
      String out =
          dag()
              .container()
              .from("alpine")
              // ERROR: cat: read error: Is a directory
              .withExec(List.of("cat", "/"))
              .stdout();
      return out;
    } catch (DaggerQueryException e) {
      return "Test pipeline failure: %s".formatted(e.getMessage());
    } catch (InterruptedException | ExecutionException e) {
      throw new RuntimeException(e);
    }
  }

🤔 But it looks like there's a 🐛 in the java sdk, because the error message is not raised as expected into the exception. I'll fix that the stderr will be visible.
In the meantime you can also handle it in the code, like that:

  @Function
  public String test() throws ExecutionException, DaggerQueryException, InterruptedException {
    var cmd =
        dag()
            .container()
            .from("alpine")
            // ERROR: cat: read error: Is a directory
            .withExec(
                List.of("cat", "/"), new Container.WithExecArguments().withExpect(ReturnType.ANY));
    if (cmd.exitCode() != 0) {
      return cmd.stderr();
    }
    return cmd.stdout();
  }

[nova elk]

Basically the error from the exec is in the error raised, but it's a bit hidden now because only the raw graphql error is available. On the above example the DaggerQueryException e looks like [Error{message=process "cat /" did not complete successfully: exit code: 1, locations=null, path=[container, from, withExec, stdout], extensions={stdout=, _type=EXEC_ERROR, exitCode=1, cmd=["cat","/"], stderr=cat: read error: Is a directory}}]
I'll see to have more useful error types so we can directly access to the stderr.

[magic rover]

Ah ok, so I should manage it from pipeline and print stdout or stderr in case of error, but from user pov the error show don't be enough to make analysis of the problem.
I mean on my example I had connectivity issue between dagger engine and nexus but the error didnt printed to stderr, for this reason I was blind and I need to go deeper with "terminal"
let me know if I clarified my point

In any case many thanks for your support

[nova elk]

Once the above issue is fixed (to have better error types) then the error (from stderr for instance) will be easier to be printed.

[magic rover]

@nova elk could I help with this activity maybe?

[nova elk]

Sure, I created this issue that summaries (or at least tries to) the problem: https://github.com/dagger/dagger/issues/10398

[magic rover]

Is clear to me, we could review the constructor of dagger query exception class or implement the to string to add that informations

[nova elk]

Maybe we could have something like dqe.asExecError() (or any equivalent more java like) and then to have access to a specific exec error type with direct access to sdtout/stderr/cmd/etc
But maybe not by default in the high level exception message. I wonder how this will look like if stderr is big.

[magic rover]

Ok so a method to extract from errors array a specific typed of error and a specific key, but in any case from pipeline perspective is necessary to catch it and use that method.
About default could be useful ad the path field and type, like a summary of errors

[magic rover]

hi @nova elk , i've opened this PR https://github.com/dagger/dagger/pull/10414 to fix it the issue let me know if is ok. thanks in advance

[magic rover]

Hi @nova elk I see your comment there are a lot of informations 🤣. In any case I think isn't necessary to split the PR.. i just need time to understand better how to proceed because unfortunately I haven't your know how 😅

[nova elk]

Yeah no problem at all. I discovered the real issue (why the stderr is not displayed, and in fact why the details are not displayed) while writing the comment 🙂
So now I have a better understanding of how we can solve it, thanks to your PR!
If you want more details do not hesitate.
I think to fix this issue will really help to write Java modules, especially when something goes wrong 👍

[magic rover]

Hi @nova elk I've updated the pr e review the previous changes based on your suggestion
Let me know if is ok or is necessary to do something

[magic rover]

good morning @nova elk i've updated the PR bevause there are issues during tests execution if you have doubt contact me! thanks in advance 😄

[nova elk]

Hi 👋
I'm sorry I'm not close to a computer for a few days, so I'll have a better look early next week.
To have the DaggerExecException extending the other looks good. But with that, do we require all the addition to throws or does the DaggerQueryException should be enough?
I haven't check (not easy on phone) but one piece is to serialise the exception in the generated entrypoint. That will help to have the error printed.

[magic rover]

hi, yes don't worry!

• about your question the tests are failing because is necessary to add it to throws (now is ok)
• your second point isn't clear to me 😅

Anyway i undestand you pov too many little changes to check from phone, we should review when you come back.

[craggy crest]

I totally forgot I didn't open this. Fixed - https://github.com/dagger/dagger/issues/10542

[magic rover]

hi @craggy crest i made the same things in this way https://github.com/robertonav20/dagger-modules/blob/main/modules/java/java-build/src/main/java/io/dagger/modules/javabuild/JavaBuild.java

[craggy crest]

Hi Rob, this is a module. How does dagger itself pull deps from a private registry? Today they are pulled from the public maven registry. Even in your module

[magic rover]

so the use case is to pull dependencies from a private registry instead of public maven registry? (enterprise use case in summary)... in my use case i used the module to publish an artifact to a private registry

[craggy crest]

Yeah I can use a module to pull and publish to my private registry. I can even do that with Go. But for java, the dagger SDK uses public maven. There's no way to use a private registry for SDK dependencies.

[nova elk]

@magic rover Thanks a lot for your work on https://github.com/dagger/dagger/pull/10414
This is a really great improvement on the Java SDK!

@magic rover

Maybe i could pick up another topic 😊
If you are interested by digging more into how dagger and the SDK is working, I have in mind something that could be the next piece of work related to (not only) errors and how the SDK communicate with the engine what is executed.
That might sound vague, but I'll write a proper ticket with all the details and what I have in mind and we can discuss that very soon.

[nova elk]

Here is what would be for me the next step on this area: https://github.com/dagger/dagger/issues/10618
This is about implementing the telemetry layer that is missing in Java. This is quite connected to the work on errors as it's about giving information about what's running in a workflow. Let me know if that's something that might interest you, or if you have any question

[magic rover]

Hi @nova elk, sorry I totally miss these messages 😅 I was very busy at work.. yes could be interesting, let me see the description

[magic rover]

Comparing the 2 traces side by side there are totally different... Honestly I saw the traces on Jaeger on my example https://github.com/robertonav20/dagger-modules/blob/main/modules/java/java-build/src/main/java/io/dagger/modules/javabuild/JavaBuild.java but I thought it was fine 😅

[magic rover]

I will try to work on it

[nova elk]

Awesome 🙂 I'll be off for a week, without access to discord, so I'll not be able to help during that time. But once I get back for sure! In the meantime, if there's anything unclear do not hesitate to post here or on the github issue.
This is a feature that could really improve the Java SDK a lot. Not on the execution side, as it's working, but on the way to understand what's going on inside, that will be super useful.

[magic rover]

Ok understood! I will keep you in touch

[magic rover]

@nova elk i made a first implementation for your ticket let me know if is ok and then we can review some details, anyway i would test directly with dagger cloud is possible during pipeline of github? or should i execute a module like this

[nova elk]

Awesome, thanks a lot. I'll have a look at the PR.
I'm not entirely sure to have understood the question. You can run dagger commands from GitHub Actions and get the traces on dagger cloud, you can use the dagger github action (see https://docs.dagger.io/ci/integrations/github-actions) for instance. What you need in fact, whatever the way the dagger command is run, is to set the right token to expose the traces.

[magic rover]

Sorry, i try to explain better my question
As we know I updated the sdk with telemetry, now I would test the result with dagger cloud. How can I do?
I'm thinking, in order to test it the module must use the updated sdk instead of official version. So there is a way to test it with GitHub action pipeline already in place?

[safe bison]

let us know if you got unblocked here @magic rover 🙏

[magic rover]

Sure, just wait @nova elk response or someone else who can answer to me 😅

I try to do my best with my free time to help the community

[nova elk]

oops, sorry I was stuck in a different problem and didn't looked at the PR

[magic rover]

Sure don't worry 😉

[nova elk]

in order to test it the module must use the updated sdk instead of official version
You can test a module using your local version of the Java SDK by referencing it using --sdk on dagger init.
But you have to follow two rules:

• the path to your local java SDK must be relative from the module path (the module you are creating)
• the path should not escape the dagger git repository.
  So for instance if you are inside dagger/dagger git repository and you want to create a test module under modules/my-java-test-module you can do:
  dagger init --sdk=../../sdk/java modules/my-java-test-module
  That way you can easily test local changes.

[magic rover]

thanks your advice it was precious to test local changes, now i'm testing it directly with dagger cloud trough dagger login command.
i see is correctly captured by sdk (OTEL_ variables and **TRACEPARENT **is configured) but when i execute the
dagger -m ./ call grep-dir --directoryArg "./src" --pattern "*
the result is different from our expectations https://dagger.cloud/dagger-demo/traces/bfce8d44166b7d0c58a9600a6f27440e?listen=cfa3506efd7a9fdc
i think grepDir span is created from new sdk but is empty, probably i miss something

[magic rover]

@nova elk have you some ideas? Anyway the PR was closed by bot

[nova elk]

I had a look some time ago, but I wasn't able to have it working, not sure why.
Maybe you can re-open a PR with the related commits rebased on top of the current main branch, it will be easier to review and understand why it wasn't working as expected.

[magic rover]

i easily open a new PR https://github.com/dagger/dagger/pull/11074 and link it to previous one, let me know if you discover the main cause

[magic rover]

ignore it, the correct PR is this https://github.com/dagger/dagger/pull/11083

[hollow roost]

Hey, starting to write my CI pipeline for my Maven project with Dagger. Currently, I'm looking for some more information on how to use the Java SDK.
Simple challenge right now. I have my multi-module Maven monorepo open in VS Code. All projects are showing up in the Java Projects Sidebar.

I cannot find a way to properly show the Dagger project to have full Java support in the IDE for Dagger development. Is there any help I can get?

[magic rover]

Hi @hollow roost, which type of support? Could you describe better your issue?

[hollow roost]

@magic rover Well "normal" Java-Development support for the IDE for developing the dagger module(s) side by side with the project sourcecode would be a good start. I didn't manage to enable code completion fox example. I would expect for the dagger module maven project to show up as a normaln maven project in vs codes "JAVA PROJECTS" component.

Maybe my current monorepo-setup with a root java project containing three modules also isn't helpful in the current situation. I was trying to set up also an equivalent dagger "top-level dagger coordinator" setup such as depicted here https://docs.dagger.io/reference/best-practices/monorepos. I failed and eventually gave up with dagger. Is there some kind of example project maybe for using the java sdk in such a situation?

[craggy crest]

I want to check in again regarding private registry support for .m2/settings.xml. We are about to start on a project (Java framework) that will eventually be used by about 2K java apps. I will be working on pipeline and delivery for this framework and I'd love to use dagger for it (currently groovy, jenkins). Without the ability to use a private registry, we can't use the Java SDK and using a different language won't work to convince the community to switch. CC @nova elk .

[nova elk]

Right now, I don't have solution to offer. But there's a current project to fix the way all SDKs can access their private dependencies (and that for me includes .m2 support).
But let me check first with @hollow locust the status as I don't think we're there yet.

[hollow locust]

Hi Nipuna, we're actively (last week) making improvements on the plumbing to make that possible (as other SDKs than Go are modules)

[hollow locust]

We were introducing git (non ssh) support. What would be your preferred auth method Nipuna ? Maybe we can prioritize that, to unlock your use-case ?

[craggy crest]

That would be awesome! I'm guessing doing a lookup for a settings.xml file. If it's in the local folder, use it, if not use the ~/.m2/settings.xml (default maven settings.xml). One caveat is, the settings.xml allows env vars (for creds). IDK how dagger would handle that. The creds thing is not a problem specific to java. Python allows it too for the index-url.

[nova elk]

I think we have a plan with @hollow locust
Probably to start with only a self contained settings.xml so reducd a bit the scope.
I'll be off next week so we'll work on it when back

[craggy crest]

Great! Yes, if I can provide a custom settings.xml that would work too

[magic rover]

Hi @hollow roost,
at the beginning i had the same doubt and difficulties with dagger on vscode but after some tries i found a way to use it and it very awesome.
Below some useful tip

• here an example project https://github.com/robertonav20/dagger-modules to create a java module and a docker compose to make e2e tests
• about code completion, try to follow these steps
  1. init module to generate sdk classes under target/generate-sources/dagger
  2. configure classpath of project and select that folder (in your use case should not be present)
  3. apply the configuration
  4. That's it, vscode java extension should manage dependency correctly

Unfortunately i didn't found a permanent way to apply this

[magic rover]

Good morning @nova elk I made all changes to PR now the trace is filled as expected, could you see the changes and give me some feedbacks? Many thanks in advance the pr after 7 days will be closed

[magic rover]

I mean closed by the bot 😂

Anyway I'm very grateful to help the community

[nova elk]

Telemetry has been added to the Java SDK. This allows to see the calls made by a java module.
The screenshot is showing a dagger call container-echo hello stdout on a default module.
On the right, you can see the previous version that only shows the call to the Java runtime. But on the left, you can see the details, displaying the different calls to container.from.withExec
That's a really great addition, 🙏 thanks @magic rover for your contribution! 🚀

[craggy crest]

Hey @nova elk , any progress on this?

[nova elk]

No yet because I was focused on something else I need to finish first, but that's our next task with @hollow locust
I'll try to have a very basic poc first, because we have questions about where to put this kind of settings (especially for java as we want it to change the way the java SDK globally act and not just for a single or a set of modules)
As soon as my current task is done I'm jumping on it.

[craggy crest]

Awesome! Thank you for the update. Our big project is about to start but I'm also going on vacation for a month so it's not super urgent 🙂 I was looking for an update.

[loud oasis]

I want to start doing some experiments with the java sdk, but we also have a requirement for private maven auth, so I'll be following the activity here closely

[loud oasis]

dagger call -m github.com/flatwound/daggerverse/cloud@main pipelines validate-change --source=. --java-version=21 --github-username=$GITHUB_PACKAGES_READ_USERNAME --github-token=env:GITHUB_TOKEN --build-tool=maven --build-docker-image
✔ connect 0.2s
✘ load module: github.com/flatwound/daggerverse/cloud@main 4.1s ERROR
┇ initializing module › ModuleSource.asModule › module SDK: load runtime › JavaSdk.moduleRuntime(
  ┆ modSource: no(digest: "xxh3:8b49f54158886132")
  ┆ introspectionJson: no(
  ┆ ┆ digest: "sha256:43b0a4f1c4c794521babdba89f153bfb899a9b1bafd1311b475ca76dc44e97e2"
  ┆ )
  ) ›
✘ ModuleSource.localContextDirectoryPath: String! 0.0s ERROR
! module source is not local

is this a limitation of the java sdk or am I doing something wrong?

[knotty gulch]

IMHO a limitation of the Java SDK (https://github.com/dagger/dagger/blob/995d367a5e8f8e4da66e421b1ab8cad89dd49489/sdk/java/runtime/main.go#L394). Does not seem to support other ModuleSourceKinds like git yet.

[still tundra]

cc @nova elk who wrote the SDK, he will know 🙂

[nova elk]

Here is the fix: https://github.com/dagger/dagger/pull/11570
I'll see to add some tests, but otherwise the fix looks like to work on my side.

[nova elk]

The PR has been merged, the main version is now able to call a module from a git source.

[craggy crest]

Any update on this @nova elk , @hollow locust ? Sorry to keep pinging. We are about to start on the new framework and this is the missing piece for me to push dagger for CI.

[hollow locust]

Hey Nipuna 👋

No worries ahah, you can ping me / us as much as you want, on the contrary

I had to pause this to improve / re-lazify the git API, which resumes the previous lazy UX of Git operations in Dagger whilst keeping the last performance improvements (+ a highway towards bigger improvements next, on Git)

It's been pretty long/tricky, but we're finally touching the end: I'm satisfied with the API, just need confirmation / feedback from the team this week.

Once done I'm resuming that work as a top-priority with a focus on Java (conjointly with Yves). Hopefully by middle of the week

[hollow locust]

You're my top priority after that 🙏

[craggy crest]

Fantastic! Thank you for the update @hollow locust !

[hollow locust]

Quick question Nipuna, apart from that: how do you like the current state of the Java SDK support ? Have you made some local tests and feel that the coverage of this SDK is enough to unblock all the corporate features do you think you'll need ?

I just wanna make sure that you'll be unlocked even in the current SDK state

[craggy crest]

That's a great question. One I don't have the answer to because I haven't tried it. I know I'll be blocked without custom registry support. All our local builds need the token set up in settings.xml so I didn't think to try it. Are you saying it may work in the current state?

unblock all the corporate features
Does the Java SDK lack functionality vs Go SDK? Majority of my use cases are to consume modules/toolchaines written in Go. Even when I have to write Java SDK code, it should be fairly simple stuff

[nova elk]

The java SDK is currently behind go/ts/py SDKs. It doesn't mean it's a lot, and it really depends on the features you need. But it's definitely a little behind. For instance I think I haven't yet added the check support.

[craggy crest]

Yes I think that's totally fine. As long as it supports basic dagger functionality it should be good!

[loud oasis]

this is what I have for a maven module, maybe it's useful?

[craggy crest]

I think this is one step ahead of where I need auth. This is for creating a Maven container with auth. I can do that with any language today. But running this requires maven dependencies and they need to come from my private registry. It's configured on my local host ~/.m2/settings.xmls file.

[loud oasis]

ah yeah if your module has private pkgs as deps then yeah this won't work

[craggy crest]

The Java SDK requires public modules too I'd assume and they need to be proxied via my private registry too. Even if I don't have private pkgs in my module.

[loud oasis]

yeah that makes sense if you use caching/proxying indeed

[loud oasis]

@nova elk : added checks support: https://github.com/dagger/dagger/pull/11704 happy to hear your thoughts 🙂

[loud oasis]

I think we have a plan with @guillaume

[proud wolf]

Have been playing around with the Java SDK. As a new Dagger user, perhaps its not the best place to start, but I really like the idea of building CI pipelines in the same language as the project it builds. On that note, most of my projects are Kotlin rather than Java, and Kotlin -- especially with DSL support (though I recognize that wouldn't exist yet) would support very nice clean build code. What is the current state of the world with respect to writing Dagger functions in Kotlin?

[nova elk]

👋 Happy to see interest on the java SDK, to not hesitate to reach out if there's some parts missing or not working as expected. Java SDK is not exactly at the same level than go/ts/py but should work for most of the cases.
Regarding Kotlin, quite some time ago I did some experiments. See #java message (also some experiments in clojure). But I haven't worked on it since then.

[craggy crest]

Java SDK is not exactly at the same level than go/ts/py but should work for most of the cases
My occasional plea for support for private registries still stands 🙂 Unfortunately, I can't consider the Java SDK as same level until it can use private registries.

[proud wolf]

Forgive my ignorance -- that code you linked is executable. How do I actually use it? Do I need to create a "dagger project" inside my project that pulls the java sdk dependencies and executes that code? So with this approach we cannot use the dagger CLI, but are rather calling the dagger service ourselves via code?

[proud wolf]

Re. private registries -- I will have this issue as well. Is it not sufficient to write out the credentials as part of the dagger chain?

[nova elk]

This code is just a quick and dirty example of how we could use the java lib in kotlin, showing the dagger objects and chaining methods.
But the lib is not currently published (you have to build it) and kotlin (or any other jvm but not java language) is not supported on module levels.
That might happen one day, but it's currently low priority.

[nova elk]

no it's not enough, because it means to access a maven config file globally so that not only the module itself but also the SDK dependencies are all going through the specified config. That's something we'd like to address soon.

[proud wolf]

Understood. As of now, the SDK dependencies are not available in maven anyway, so this is irrelevant for now right? I'm not sure I understand the entries in the pom.xml that reference the non-existent dependencies. Presumably that context becomes available inside the container when dagger runs?

[nova elk]

When we are running modules, the dependencies are generated on the fly because they depend on the configuration (for instance we need to generate types and functions based on the other dependent modules)
So at least a part of those libs can only be dynamic and not pushed anyway.
We have ideas to improve that, but currently that's how it works: by generating all the dependencies inside the Java SDK.

[knotty gulch]

I see debugger attachment being part of the overall efforts in establishing the Java SDK https://github.com/dagger/dagger/issues/9486. Can you elaborate a bit how this would work? Is this remote debugging (sth like java -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 -jar deine-app.jar) ?

[nova elk]

I have not made any progress on this for quite some time... But yes the idea is to explore how we could expose remote ports to attach a debugger. But nothing done on this area for now.

[craggy crest]

Hey @hollow locust ! I'm desperate to show off dagger to a whole bunch of Java devs but I can't without private registry support 🙁 Showing it in Go doesn't have the same effect!

[nova elk]

Finally some good news!
I just pushed a PoC branch here https://github.com/eunomie/dagger/tree/dagger-java-mvn-settings
This is a custom version of dagger that loads the ~/.m2/settings.xml file inside the java container used by the Java SDK to build modules written in Java. No modification is required on Java modules, it's all transparent.

⚠️ This is a quick and dirty version, there's probably a lot of things to improve, clean, test (and it's mostly vibe coded).

But from the manual tests I've made, it's working! I have a second machine with a reposilite instance, and all the maven downloads are going through it.

• no configuration, if there's a ~/.m2/settings.xml it will be loaded
• self contained: the ~/.m2/settings.xml is the only piece used, no env var, no interpolation, nothing. The file must contain all the needed information (so including secrets if any)

The goal of this branch is to act as a quick and dirty PoC so you (and any other people of course) can test it, see how that works, and unblock the demo of dagger using java modules and private registries. And then, we can iterate on it, make it a real feature, take your feedback on it, etc, up to the point we can include it in a release.

[craggy crest]

love it! How do I test it?

[still tundra]

Nipuna right now https://giphy.com/gifs/cbc-alexis-selfish-for-me-JRUbQVADfqzrFnOkZr

[craggy crest]

haha! for sure! loved that show btw

[still tundra]

Same 🙂

[nova elk]

you need to build it on your machine
If you clone this branch, then:

• ./hack/build will build it and start a dev dagger version
• then you replace dagger ... by ./hack/with-dev dagger ... to call this dev engine
  That's an easy way to test, as you need to have access to your local ~/.m2/settings.xml file. I'll see if we can also use the playground

[nova elk]

An other way, only using dagger commands:

$ dagger -m "https://github.com/eunomie/dagger@dagger-java-mvn-settings" \
  call engine-dev playground \
  with-file --source=~/.m2/settings.xml --path=/root/.m2/settings.xml \
  terminal

This will build a dagger engine in a container, based on this branch, and open a terminal inside. A dagger in dagger. Easy as it only requires dagger, no need to build and wrap commands.
And this engine also contains the ~/.m2/settings.xml copied inside.
Then, inside the terminal, you have access to a dagger binary so you can for instance:

$ dagger init --sdk=https://github.com/eunomie/dagger/sdk/java@dagger-java-mvn-settings my-module
$ cd my-module
$ dagger functions

All that should hit the registry mentioned in your ~/.m2/settings.xml.
Just note the sdk pointing to this branch as java sdk is not embedded like go/python/typescript SDKs.

[craggy crest]

I was running ./hack/dev (./hack/build isn't there) and running into issues. Let me try the second way. I think that may work better

[hollow locust]

Thanks @nova elk for pushing it . Please use it with only modules that you control / have checked, as any module will potentially have access to those file. Until we find the right way to express the access control, this should unlock you 👼

[craggy crest]

man! I can't run it this way. It's trying to pull directly from public container registries. We are blocked 🙂

[hollow locust]

How can you run it ? What do you need mounted in the engine to pull from your private registries

[craggy crest]

If you could build and upload it to a container registry, I can try running it that way. Does this change require a client change too?

[craggy crest]

I need to proxy containers via our private registry. I guess I could set up registry mirrors hmm

[craggy crest]

I am going to build the engine from my personal machine and pull it. Do I have to build the CLI for this too? or just the engine is enough?

[hollow locust]

I don't think you'll have to build the CLI, but Yves told me privately that he's gonna push the engine for you on a registry tomorrow

[craggy crest]

I can just extract it from the engine image right? It's bundled?

[craggy crest]

That's awesome! I couldn't wait as I'm impatient 😄 I have the engine running now. Built it on my personal

[hollow locust]

if you do: ./hack/dev locally you'll have it in ./bin i guess

[craggy crest]

well, isn't it bundled in the engine now? I can easily get the container pulled. It's harder to get a binary

[hollow locust]

yes it is you're right:

 docker exec -it dagger-engine.dev bash
bash-5.3# dagger version
dagger v0.19.12-20260224183130-dev-4fa7278352ca (unix:///run/dagger/engine.sock) linux/amd64

[craggy crest]

I ran into the first snag..

maven:3.9.9-eclipse-temurin-21-alpine@sha256:4cbb8bf76c46b97e028998f2486ed014759a8e932480431039bdb93dffe6813e: failed to resolve source metadata for docker.io/library/maven:3.9.9-eclipse-
  temurin-21-alpine@sha256:4cbb8bf76c46b97e028998f2486ed014759a8e932480431039bdb93dffe6813e: unexpected status from HEAD request to https://registry-
  1.docker.io/v2/library/maven/manifests/sha256:4cbb8bf76c46b97e028998f2486ed014759a8e932480431039bdb93dffe6813e: 403 Forbidden

So it's not like Go, It's not bundled and it tries to pull stuff. But my registry mirror's aren't set

I'm getting close! Set the registry mirrors and got past that

hm I see this ModuleSource@xxh3:2e388b3cdeaa2127.hostConfigFile(name: "maven-settings"). I'm trying to init a new dagger module but it's failing when trying to pull dependencies. It's trying to pull from maven central so that indicates it's not using my settings.xml

[craggy crest]

Got it to work!! 🎉 (needed both the CLI and engine to match)

However...

Doing anything with the Java SDK is super slow. It took almost 7-8 mins to init a module. Takes like 4+ mins to do a dagger install and about 1+ mins to run anything for the first time (it's snappy after cache). I expected maven and java to be a bit slow but not by this much. Is that your experience too @nova elk, @hollow locust ? This is only local machine. In CI it will be even slower as we start fresh

[craggy crest]

Sometimes it gets stuck in module SDK: load runtime for 5+ mins 🙁

[nova elk]

I don't have the same kind of numbers on my side.
Of course all that will also depend on the maven registry/proxy/cache you're hitting (and same thing for the docker proxy to fetch maven and java images).
And of course the very first run that will download all the needed jars will be slower. Once they are in your local proxy/cache it should be way quicker as it's local.
At least that's how that works on my side, I have a reposilite instance on a small Nuc.
So if I already did it once (to have the jars in reposilite) here are my numbers on a completely fresh engine (using the playground command above on a dagger engine where I prune all the cache)

• the dagger init takes 47s (16s to load the SDK from github and my specific branch, 31s to generate the module, codegen, etc)
• a dagger functions in the created module took 18s to load the module
• a dagger call container-echo --string-arg plop stdout took less than 4s, 0.9s to load the module, 2.6s running the container-echo function, pulling the container, etc

If you're completely fresh, your maven proxy/cache and your docker proxy/cache has nothing inside, yes it can be slower, but this overhead is not really related to dagger. If you were building the java sdk using maven for instance your very first call will be similar, and all the others will be faster, as expected. And this overhead depends on the speed of your infra components and the associated bandwith.

That said, we had some ideas to improve the Java SDK performances, generate less code on the fly, etc. But for now it has not been prioritised due to the low interest and feedback on this particular SDK. But that can change for sure.

[craggy crest]

I am definitely interested in performance for sure because I want my consumers to feel the same speed as using dagger with Go. That being said, is there a reason you aren't using threading for the maven calls? I don't see -T being added to any of the calls the SDK makes. I'd imagine at a minimum you could do -T 1C. We do that for most of our mvn commands

[nova elk]

no specific reason, just never added. -> https://github.com/dagger/dagger/pull/11925
I force pushed https://github.com/eunomie/dagger/tree/dagger-java-mvn-settings rebased on main (after the 0.20 release) and with -T 1C to see if that's better.
On a fresh (no cache) engine I have 37s for a dagger init (10s less than previously, but I haven't run a lot of tests, at least it shouldn't be bad to have it)

[craggy crest]

Just did another init (not even empty cache) before pulling your -T changes and got generatedContextDirectory: Directory! 6m34s

I am seeing that it's spending time pulling layers from the temurin maven image for example a lot. Would pinning those images to SHA help?

[nova elk]

Is it possible you share some traces (or via DM of course)? To see what aspect is taking so much time.
Or run it with -vvvv so it will expand a bunch of traces

[nova elk]

they are already pinned

[craggy crest]

Hmm, then it's strange that the engine is looking for updates

[nova elk]

and it should be cached after the first run

[frigid shell]

Hey !
I noticed* that the load module was quite long when I had too many files in the source folder.
For exemple a node_modules. Make sure to exclude them from your dagger functions

[nova elk]

It shouldn't, here is what I have in my traces.
It takes 1.9s, that is too much, but it's not minutes. And it's only to resolve the image ref.
That's something we would like to remove, and I know there was some work on lock files that might be related.
But that shouldn't take minutes. If that takes minutes it means the communication between the engine and the registry (through an image proxy?) is slow and that the image is not correctly cached.

[craggy crest]

Multi threaded sped it up a bit. dagger init took 5m24s still. After doing a dagger core engine local-cache prune

[nova elk]

Do you have any trace or logs (-vvvv for instance) to see where the time is lost? If you have a local maven cache and a local docker cache that should not be that much. But impossible to say without to see more info.

[craggy crest]

First call of dagger functions similar to last time `1m54s1

[craggy crest]

I can run it with -E and get you the timings. When I observed, most of the time is spent during mvn calls.

[nova elk]

But once you did it one, all the deps should have been fetch and made available in your local/internal cache, no? It could be interesting to see if it's to fetch the jar between dagger and your registry that is slow of if it's something else, like the build. If the traces are on cloud it should be doable to investigate all the steps

[craggy crest]

We don't use cloud 🙁 can't

[still tundra]

@craggy crest feels like a good time to ask: want to take the lead on this SDK?

With the latest coding models it becomes way more achievable

[craggy crest]

I don't mind contributing where needed! "taking lead" depends on what it entails 🙂 First I am trying to evaluate if it's something we can recommend to devs. So far, speed is the issue. It may be because our proxies and package managers are slow with pulls but Go doesn't have that problem

[hollow locust]

With Yves, we could totally do sessions with you to find the bottlenecks and have you in the loop ? After all it's only code ahah, dang + java should maybe be ways to make it faster ?

[craggy crest]

It's codegen that's taking longest (This time it took 7m+). I can't copy from the TUI when I do -E (@nocturne crane is that a bug?) so I'm taking some screenshots. One of those steps is pulling the docker image. This is from a clean cache. So the slowness is mostly the fault of vpn+internet+artifact-proxy. Hence why you aren't seeing it on your side. @nova elk your remote artifact storage is local so that would be much faster.

[nocturne crane]

try holding shift + click+drag to select - it's because the TUI takes over the mouse

[craggy crest]

Something of note: I see multiple mvn versions:set and mvn clean compiles so, maybe opportunity to consolidate.

[craggy crest]

oh that works! might help to add a hint (next to E).

[hollow locust]

What would be nice, maybe, is to have a top-level spec-issue that basically lists the current state of the Java SDK (according to you) and what's needed for your use-case: the minimal shippable version for internal proposal / deployment, all the bottlenecks

If together we fix the codegen, would that be feature-enough ? 👀

[craggy crest]

Potentially! I'm not looking to do anything crazy with the Java SDK. Hopefully with workspaces/modulesV2, end users don't need to write any dagger. It'll hopefully be for advanced users that want to branch out of our prescribed pipelines. My main daggerverse is in Go.

[hollow locust]

I see, makes sense ! 😇🙏

[craggy crest]

Besides the slowness, the actual settings.xml ingestion, from a functionality POV, seems good to ship! IDK if it still needs more polish. Potentially an explicit setting in the dagger.json for the default location?

[nova elk]

So the slowness is mostly the fault of vpn+internet+artifact-proxy
Ok, but it means even if you just pull the same maven components from your own machine, it will be slow the same way, no?
If to pull maven dependencies are super slow, maybe some part of dagger can make it faster but it will always be once cache is filled up, never before.
Or am I missing something else?

[nova elk]

Yeah, it was a good idea at start, but I wanted to get ride of this. I have a version somewhere on a branch, but it's not yet in main

[craggy crest]

Ok, but it means even if you just pull the same maven components from your own machine, it will be slow the same way, no?
If to pull maven dependencies are super slow, maybe some part of dagger can make it faster but it will always be once cache is filled up, never before.
Typically, it's not that slow. There is a cost to starting up a mvn command. Like it always hangs for a sec or two but after that, it's usually pretty fast. If we can somehow reduce the number of mvn commands that are run for bootstrapping, that may improve things.

Also, is there a way for me to run these same commands outside of dagger? I can observe the speed diff that way.

[nova elk]

not exactly the same but you have the java projects under sdk/java. And from there you can mvn clean install for instance.
I removed my ~/.m2/repository and here are my numbers (but I'm not at home, so I pulled directly from maven central)

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for dagger-sdk-parent 0.20.0:
[INFO]
[INFO] dagger-sdk-parent .................................. SUCCESS [  4.746 s]
[INFO] dagger-codegen-maven-plugin ........................ SUCCESS [ 19.946 s]
[INFO] dagger-java-sdk .................................... SUCCESS [ 26.570 s]
[INFO] dagger-java-samples ................................ SUCCESS [  0.658 s]
[INFO] dagger-java-annotation-processor ................... SUCCESS [  4.315 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  56.722 s
[INFO] Finished at: 2026-03-02T09:57:23-08:00
[INFO] ------------------------------------------------------------------------

[craggy crest]

Where exactly did you run the command @nova elk ? I tried one of the examples but it complains that the sdk is missing.

[nova elk]

inside sdk/java, I just ran mvn clean install

[craggy crest]

It may be because you rlocal cache already has the dagger-codegen-maven-plugin. I had to run that first

[nova elk]

You have to build and install first so it's made available on your machine

How much time did it took to run it compared to the 20s above?

[craggy crest]

oooh.. just inside sdk/java

running inside sdk/java now

stuck for a long time here
[INFO] --- dagger-codegen:1.0.0-SNAPSHOT:codegen (default) @ dagger-java-sdk ---
[INFO] Querying local dagger CLI for schema (version=0.20.0)

[nova elk]

Is dagger running? It needs to access dagger to fetch the schema
That's what I have id dagger is not running, but it failed immediately

[INFO] dagger-java-sdk .................................... FAILURE [  9.913 s]
[INFO] dagger-java-samples ................................ SKIPPED
[INFO] dagger-java-annotation-processor ................... SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  27.472 s
[INFO] Finished at: 2026-03-02T10:44:47-08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal io.dagger:dagger-codegen-maven-plugin:0.20.0:codegen (default) on project dagger-java-sdk: Execution default of goal io.dagger:dagger-codegen-maven-plugin:0.20.0:codegen failed: Command dagger query -s -M exited with code 1 -> [Help 1]

And it's trying to build dagger-codegen:0.20.0:codegen , not dagger-codegen:1.0.0-SNAPSHOT:codegen. But maybe because I'm running that from main of dagger/dagger? Or you have an old version?

[INFO] --- dagger-codegen:0.20.0:codegen (default) @ dagger-java-sdk ---
[INFO] Querying local dagger CLI for schema (version=0.20.0)
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for dagger-sdk-parent 0.20.0:

[craggy crest]

I tried running it in your fork main. What do you mean by "Is dagger running"? I do have 0.20.0 running but I'm not in the dagger/dagger main

[nova elk]

Can you try from the main of dagger/dagger and not my fork? To have the same one running so it can fetch the schema. Just to validate that works

[craggy crest]

This is my timings from scratch - mvn clean install -DskipTests -T1C -Dmaven.repo.local=.m2

Subsequent run

[nova elk]

This is really experimental, but that's some ideas I'm exploring: #maintainers message
This should make the Java SDK way faster

[craggy crest]

@nova elk has there been any progress with this? Any thoughts on getting the private module support in?

[nova elk]

On private registry no. My understanding is even with it, the performances would be quite bad because of the slowness of proxies/remote registries.
We are working on performances aspects, that I think will bring a lot of benefits. And many other changes on the SDKs. I think it will be more beneficial to go back to the private registry thing after that.

[craggy crest]

That's fair. I'm thinking i'll use dang in the meantime for Java modules/apps. I asked a question on the other channel about dang but didn't get an answer. Can you shed some light on it? #daggernauts message

[still rain]

Hi everyone, I wanted to start using Dagger for my multirepo AI Factory, but I'd use it with Kotlin. So the Java SDK should work.

However, I have a hard time to find a 'right' library. On Maven it seems to be https://mvnrepository.com/artifact/io.lgtd/dagger-java-sdk/0.16.3-010101000000-dev-f003e7bcf025, and I found https://daggerverse.dev/mod/github.com/dagger/dagger/toolchains/java-sdk-dev@1d5dabfd4f6bea9b57968587b28d48b03098140e with updates from yesterday. As well this here https://github.com/dagger/dagger/tree/main/sdk/java and this https://github.com/dagger/dagger/discussions/12131#discussioncomment-16300186 or should I just look at dang? (https://github.com/vito/dang)

Where should I start? 😅

Update: ok, found this https://github.com/dagger/dagger/issues/9486 I'll look into dang then.

[nova elk]

👋
You can use the java sdk to create modules, just dagger init --sdk=java my-module and you'll get started.
It doesn't work with Kotlin. The libraries themselves could, but they are not (yet) published and the modules are embeding more than just the libraries, they also contain the build steps that is different for kotlin and java.
I'm preparing some improvements on the java modules, including better performances et better integration with IDE/local dev, but it's not yet ready.

That said, of course you can use dang, and dang is really a good choice for many modules (we're migrating multiple modules to dang). Java (or go, ts, etc) will still offer more features on the language side.