typescript

[quiet knoll]

👋 @grave flare

[grave flare]

Thank you! 😁

[quiet knoll]

https://tenor.com/view/nod-smile-gif-9743727

first gif that came up for "node". seemed appropriate.

[grave flare]

Didn't get as much time to try out testing frameworks, but man I'm not a fan of Jest. Getting it to work with both TypeScript and ESM is such a pain. I feel like for what I've so far done with Dagger, I haven't found a need for mocks, so playwright has worked really well. Testing connect is interesting because it's not a unit test but it's also not an E2E like how playwright defines E2E. It's kind of more of an integration test, kind of an E2E in its own way?

Both vitest and playwright use vite under the hood so it's made setting up testing incredibly easy, but vitest's isolation is file based while playwright's is on every test. Vitest also has this weird issue of hanging when ran inside of dagger run. Playwright works like a dream wrapped in dagger run.

I bet as I explore using Dagger in crazier ways I'll have a need for mocking and I'll have to experiment with mocking within playwright.

[grave flare]

I'm currently testing the native node test runner and so far it's working out REALLY well. https://nodejs.org/api/test.html

[grave flare]

The node test runner is so far a great option for general testing, but it shares a similar problem to vitest where isolation is by file, making the usage of _EXPERIMENTAL_DAGGER_JOURNAL to output the journal way more tricky than Playwright. It's also not as mature as vitest, so getting things like the full test name including the suite is more tricky and getting things like the test path with file name is impossible. Unlike vitest, it works great with dagger run just like Playwright.

I'm not sure what's up with vitest and dagger run, I wish it worked better.

That and if the journal events could be exposed programmatically like LogOutput, it would make things way easier for diagnostics on why a test failed. This can kind of be achieved by running all tests under dagger run and setting _EXPERIMENTAL_DAGGER_JOURNAL at the dagger run level, but then it's one massive journal instead of broken up smaller journals.

[grave flare]

Doing some real world testing with using Playwright test as a unit testing library, and so far it's flakey 😕 going to try to switch to the native node test runner to see if it's any better.

[grave flare]

Working better with node test runner, but some times getting strange hangs when running it without dagger run. Very interesting.

[grave flare]

Not sure what has changed, but trying out vitest again seems to work with dagger run.

[hexed tree]

Any plans to ditch node and replace with Bun?

[bright mountain]

Well we always want Dagger + Node to work great together. But I guess you mean for the Typescript SDK.

That's a question for @indigo sinew 🙂

One step we are taking with the upcoming Zenith release (0.10) is actually calling it the "Typescript SDK", rather than the "Node SDK". The rename doesn't remove Node under the hood, but it makes it at least semantically possible.

(The reason for the rename is that Javascript will no longer be a first-class citizen going forward. When developing Dagger Functions, the lack of types becomes a major handicap)

[hexed tree]

Using node for module execution seems painful. My npm install for a module that only uses the Dagger SDK took 2 mins

With bun it takes seconds.

Bun also doesn't need Typescript SDK or ts-node or transpilation.

I'd like to see Bun used, even though end users are free to use Node for their own modules

[indigo sinew]

It's in progress: https://github.com/dagger/dagger/pull/6730

With that, we'll detect the runtime (Node or Bun for now) and will execut the module in your favorite 😛
So normally, no more npm install if you want to use bun

Because Bun & Node code is exactly the same, it's possible to just replace the runtime in a few lines, which is not possible for now with Deno because the code isn't 100% compatible specially regarding import

So once this PR is merge, you should be able to try running bun as your TsSDK runtime 😉

Even if we'll always use node by default, it will be super easy to change (just a field in the package.json)
And for more context: see https://github.com/dagger/dagger/issues/6591

[hexed tree]

Awesome

😀

Subscribed to issue

[earnest tusk]

Hey folks,
I have some noob questions, so sorry for that.
I would like to create one module for node.js that contains several steps to build and test.

There is an already SDK which seems great but my noob questions are;

• If i use Dagger module, does it mean no need to have Dockerfile anymore?
• Can I able to pass a parameter to this module? What I meant, i would like to create one Module such as below, but i want rest of the engineering teams can also able to use this module hence they probably need to pass different image while calling the module that we have on top of this.
• How can I extend the container functionality here? For example, can i add custom functions like scan and sign the image?

import { connect, Client } from "@dagger.io/dagger"

// initialize Dagger client
connect(
  async (client: Client) => {
    // get Node image
    // get Node version
    const node = client.container().from("node:16").withExec(["node", "-v"])

    // execute
    const version = await node.stdout()

    // print output
    console.log("Hello from Dagger and Node " + version)
  },
  { LogOutput: process.stderr },
)

[earnest tusk]

You're right, sorry for that Scott. I just could not decide which channel should I use, but I deleted another one.

[fathom obsidian]

Ok. I'll delete my post too. 🙂

Hey folks,

[indigo sinew]

@glad kraken I created an issue related to my research related to the TypeScript performance improvement: https://github.com/dagger/dagger/issues/7093

I'll report what you said in your PR, hopefully we'll find way to make it faster

[true nova]

Hello! I have a kinda noob question about running Dagger from a Typescript project (Playwright in my case): is there a way to retrieve the engine logs when running a Dagger command? For example: const res = await dag.container().from("alpine:latest").withExec(["echo", "-n", "Hello world"]).stdout() returns "Hello world" as expected, but I would also like to somehow configure the Client to log what it's doing (that would be equivalent to the stderr output of the dagger --progress plain call container-echo --string-arg 'Hello World' stdout command)

[maiden eagle]

How do you use tsx with .mts files in GitHub Actions? When I use args: tsx foo.mts I got "tsx": executable file not found in $PATH

[stoic tulip]

👋 what are you trying to do with GitHub Actions?

[maiden eagle]

run the CI pipeline

[white prawn]

Hi - I am seeing odd behaviour both locally and in a GitHub action. Walked away and came back to the computer and it's suddenly occurring in every environment.

Error: response from query: input: node resolve: call constructor: process "tsx --no-deprecation --tsconfig /src/me-function/tsconfig.json /src/me-function/src/__dagger.entrypoint.ts" did not complete successfully: exit code: 1

Stderr:
node:internal/process/promises:289
            triggerUncaughtException(err, true /* fromPromise */);
            ^
TypeError [ERR_UNKNOWN_FILE_EXTENSION]: Unknown file extension ".ts" for /src/me-function/src/index.ts
    at Object.getFileProtocolModuleFormat [as file:] (node:internal/modules/esm/get_format:160:9)
    at defaultGetFormat (node:internal/modules/esm/get_format:203:36)
    at defaultLoad (node:internal/modules/esm/load:143:22)
    at async nextLoad (node:internal/modules/esm/hooks:749:22)
    at async x (file:///usr/local/lib/node_modules/tsx/dist/esm/index.mjs?1714856166980:2:1744)
    at async nextLoad (node:internal/modules/esm/hooks:749:22)
    at async Hooks.load (node:internal/modules/esm/hooks:382:20)
    at async handleMessage (node:internal/modules/esm/worker:199:18) {
  code: 'ERR_UNKNOWN_FILE_EXTENSION'
}

Node.js v21.3.0

[white prawn]

I have fully documented the above error in a GitHub Issue along with a repository that reproduces the error: https://github.com/dagger/dagger/issues/7284

[indigo sinew]

Hey, thanks for documenting the issue, I tried on my side and wasn't able to reproduce it locally.
https://github.com/dagger/dagger/issues/7284

[white prawn]

Thank you @indigo sinew for your hand with this one! I replied on GitHub but looks like everything is working now, phew!

I'm a bit confused as to how the error occurred and rectified itself, am I right in thinking that the Dagger Engine always pulls the latest version of the SDK?

[solar shuttle]

No. When using modules, the SDK is bundled in the engine container when it's released. It's exposed to you when you do dagger develop (during codegen), under the sdk directory, it's not pulled from the package registry. In other words, the SDK version is always the same version as the CLI (dagger version). Again, this is only when using modules since they run in a sandboxed/containerized environment.

[indigo sinew]

Thanks for your reply! I'm glad your problem is fixed

[white prawn]

Thank you Helder for the explanation! That's good, with that best guess scratched out I'm now left scratching my head to figure out how this occurred!

[indigo sinew]

@glad kraken @half violet I bumped the linter version with several changes cause the previous one seamed to be broken
https://github.com/dagger/dagger/pull/7289

[indigo sinew]

@solar shuttle I fixed the issue you mentionned in the doc custom type here:
https://github.com/dagger/dagger/pull/7381

[indigo sinew]

@solar shuttle Btw, I'm going to post update on my research to improve SDK performances:
https://github.com/dagger/dagger/pull/7096

As a summary in case you missed it:

• Using yarn to install dependencies, I could reduce (almost every time) the initialization step by 2
• By reorganizing the sdk setup inside the Base function, the SDK benefits more from the cache specially if the user only updates the module code
• By using --production, we can skip the installation of unecessary dependency, but I think this is also what create failures in the CI, I'm trying to figure why.
• By using dagger instruction instead of shell's, we benefit more from the cache.

I also add some simplification in the runtime and added new cache keys for yarn.

Things I tried but didn't worked:

• Using yarn to install tsx as a global dependency seems to make things slower.
• Using pnpm completly break the setup for obscure reason.
• I had the idea to add the ts sdk node modules in the dagger binary as a volume to skip the install part but it would double the dagger size.
• Using bun to install SDK dependencies (even in bun), it could have worked but we still need to use npm later, and the installation makes it slower.
• Use bun --production: make the setup fails because the lockfile isn't frozen, I took time to search why but didn't found a fix so I put my focused into more impactant optimization.
• Check for way to optimize the install itself, I didn't find anything relevant, we need to download these dependencies so we're quite stuck on this.
• Cache the node_modules directory -> did not really improved performances, this directory is really big

Things I want to do next

• Improve helpers scripts to be faster, they take 1-2 seconds to run which is not that much but 1 + 2 + 3 + 4 = 10 extra seconds
• Check for node engine configuration, that might help make things faster when the code run
• Fix the CI issue with the new setup, it works with my test but not in the CI, I need to figure out why.

/cc @gloomy shale @glad kraken

[indigo sinew]

@half violet CI passes, I had to remove the yarn cache to fix it though so I'm not happy about it, I need to understand more about the yarn cache collision that happens in the CI, as I said in my comment, I have a simple fast fix in mind: specific cache key for each module but I'm not sure we want to do that.
https://github.com/dagger/dagger/pull/7096

We should be able to get that PR merge by the next release.

@solar shuttle I'll take care of your suggestion when I have more bandwidth, I want to allocate time on contextual module

I published a small conclusion of this PR to keep track of what have been done and what I could do next: https://github.com/dagger/dagger/pull/7096#issuecomment-2123537679

@drowsy oracle I also fixed an issue you raised few time ago: https://github.com/dagger/dagger/pull/7434
To review by @solar shuttle @glad kraken or yourself if that makes sense to you

[indigo sinew]

@half violet Let me know if you have any bandwidth to check on that today 😄
It's mergeable as soon as I get an approval

[indigo sinew]

https://github.com/dagger/dagger/pull/7096 has been merged and will be part of the next release!

/cc @glad kraken @half violet @gloomy shale @solar shuttle

[indigo sinew]

@solar shuttle @uneven harness I did some investigation on using a TypeScript query builder instead of our custom's and I'm not satisfied by the solution proposed by the library.

I put some details there: https://github.com/dagger/dagger/issues/5609#issuecomment-2125790089 and would love your inputs on it.

[indigo sinew]

@solar shuttle I would like to work on supporting interface in Typescript, can I start to work on an implementation soon or the overall interface design will be redefined?

[solar shuttle]

Yeah, you can start working on it.

[indigo sinew]

@solar shuttle @uneven harness
Hey!

I'm quite stuck on the return value logic for interfaces...
What is the actual value that is expect by Dagger to be returned by a function that return an interface?

For example:

import { func, object, daggerInterface } from "@dagger.io/dagger"


/**
 * Image interface
 */
@daggerInterface()
export abstract class Image {
  /**
   * Return the type of the image.
   */
  abstract type(): string
}

@object()
export class PNG implements Image {
  @func()
  type(): string {
    return "png"
  }
}

@object()
export class JPEG implements Image {
  @func()
  type(): string {
    return "jpeg"
  }
}

/**
 * HelloWorld class
 */
@object()
export class Interfaces {
  @func()
  createImage(type: string): Image {
    switch (type) {
    case "png":
      return new PNG()
    case "jpeg":
      return new JPEG()
    default:
      throw new Error("Unknown image type")  
    }
  }

  @func()
  getType(image: Image): string {
    return image.type()
  }
}

I keep getting the error:

failed to convert return value: unexpected interface value type for conversion from sdk result map[string]interface {}: map[]

I tried to figure out for the whole day what I'm supposed to return but I do not get it?
My marshaller return a PNG object (same as if we were returning PNG) so it supposed to match the image interface but I don't know :/

I created a Draft PR, still WIP but I'm stuck on this part: https://github.com/dagger/dagger/pull/7475
I would like to get some help if that's possible

[indigo sinew]

@solar shuttle Hey, could you please review this PR when you have a moment?
It improve the TS SDK DX by constructing the class if a parameter (or a field) is an object of the module itself: https://github.com/dagger/dagger/pull/7477

This fixes: https://github.com/dagger/dagger/issues/7446

[solar shuttle]

Yeah, I'll take a look.

[edgy hollow]

👋 @indigo sinew I just accidentally returned Promise<String> instead of Promise<string> (lowercase S) in a function and got a little surprise that dagger functions errors if that's the case even though my TS linter doesn't seem to complain. Does that make sense?

[kindred umbra]

Promise<String> is valid typscript since String is the constructor function (class) for strings. So returning a Promise<string> from a function annotated with Promise<String> will work with typescript.

[edgy hollow]

👍 . that's what I imagined.. seems like we're not handling that case in our code generation part 🙏

[indigo sinew]

Yes we're not handling that case :/

The String, Number, Boolean and types are reference the interface that and string is a conveniance alias that hold the value.
So I'm not sure how to handle that properly because it's a bit more complex than it seems.

I could make a reference table but then what about other primitive types like Object?

Based on this playground, the underlying type is string, I need to think about it, could you open an issue about that please?

We got a super useful answer here too: https://stackoverflow.com/a/14727461/14191060

@glad kraken Hey 😄 may I can get a review on this one so we can merge it: https://github.com/dagger/dagger/pull/7481 ?

@rotund ember I remember you're the one creating an issue related to LTS so here is it ^

[half condor]

Dagger <@&1113692044158836736> users - if you missed the latest SDK updates from the last community call, check this recording out. https://discord.com/channels/707636530424053791/1245142923381702779

[quasi cliff]

I don't think I use typescript. Where do I unsubscribe?

[icy pine]

at the top of the server's channel list, you should see a Channels & Roles button
there you should be able to remove and add the languagues you are interested in

[maiden eagle]

Right click on the channel

[kindred umbra]

I commented on the PR as well, but Node 22 doesn't go into Active LTS until the end of October

[kindred umbra]

string isn't really a "connivence allias" for string.
String is the constructor function, and exists in javascript, whereas string only exists in typescript.
... Although I kinda understand why you called it that though

[indigo sinew]

Yes it's the current, so I thought it was okay to use it, what do you think?

[indigo sinew]

Yeah I tried to keep it simple

[kindred umbra]

According to node,

Production applications should only use Active LTS or Maintenance LTS releases.

[indigo sinew]

Right! Okay I change it to 20 tomorrow

[indigo sinew]

@kindred umbra I merged https://github.com/dagger/dagger/pull/7477, let me know if you hit any issue with the prototype loading anymore.

The feature should be available in the next release 😉

[edgy hollow]

@indigo sinew having an regression issue with typescript module initialization in v0.11.5:
https://play.dagger.cloud/playground/5b-HAgNycwD

^

not sure if it has been reported. That works in v0.11.4 and seems to only affect typescript modules for what I've tested:
https://play.dagger.cloud/playground/3Z2n-n7EZsk
^ this works

[indigo sinew]

What's your TypeScript module link?

[edgy hollow]

https://github.com/marcosnils/daggerverse/tree/main/gptools/examples/typescript

[indigo sinew]

Ok I need to check too

[edgy hollow]

it only fails if you try to load the module as I do on the playground snippet

if you do dagger develop on the module root page, it works

[indigo sinew]

Oh that's weird

What about dagger functions?

Or a simple dagger call from the CLI?

Because the logs from the playground are not that helpful

[edgy hollow]

I can give you a Dagger pipeline to repro this. 1 sec

[indigo sinew]

Would love that

[edgy hollow]

type Lala struct{}

func (m *Lala) Test() error {
    _, err := dag.Git("github.com/marcosnils/daggerverse").
        Branch("main").Tree().
        Directory("gptools").
        AsModule(DirectoryAsModuleOpts{SourceRootPath: "examples/typescript"}).
        ID(context.Background())
    return err
}

if you run that in v0.11.4, that works 🙏

[indigo sinew]

Ty

I'm checking that when I can, I first need to solve another issue

[edgy hollow]

sure, no rush for now

[indigo sinew]

@edgy hollow Are you sure it's related to TypeScript itself? The error message isn't triggered by the runtime nor the TS SDK

[edgy hollow]

Well.. it doesn't happen if you change the path to examples/go

So the fact that it's not a go module matters

Not necessarily related to typescript specific issue though

[indigo sinew]

Your go example is still under v0.11.4

Could you update it to v0.11.5 and see?

[indigo sinew]

@edgy hollow In case you missed my answer

[edgy hollow]

IIRC it worked also in 0.11.5. Let me double check

[edgy hollow]

Yep, works in v0.11.5 https://play.dagger.cloud/playground/tltmIdaXVd7

Just updated the dagger.json @indigo sinew

I'll check if it happens with python tomorrow

[indigo sinew]

Ohhh that's funny

So the codegen only failed on TypeScript

That's sadly strange and sad

I'll dig into that tomorrow, I have a limited timebox though because other features development + podcast recording.

[kindred umbra]

Can we make the package manager be configurable?
I noticed that it recently changed from the latest version of npm, to the classic version of yarn.
I also can't update the version of yarn it uses because yarn install --production doesn't work in latest versions

[indigo sinew]

https://github.com/dagger/dagger/pull/7518 should fix this issue

And yes I want to make it configurable but it's not has easy as it looks, because each major version have breaking changes, so this PR handle installations globally but isn't perfect, it will work but not using the package manager selected (for now)

See https://github.com/dagger/dagger/issues/7433#issuecomment-2141495905 for the latest update

[kindred umbra]

I know that its not easy, which is why I thought that you were always using npm.... I was a little surprised when you switched it

[edgy hollow]

@indigo sinew I'm coming back at this.

Just tested with v0.11.6 and the issue is still there. One additional important detail is that the issue seems to manifest only when loading typescript modules:
https://play.dagger.cloud/playground/YEjXOthcF8r

^ if you replace examples/typescript with examples/go or examples/python it works in both cases

[indigo sinew]

Hmm strange okay! Thanks for the update I’ll check that

[indigo sinew]

On it, reproducable locally, I think the error comes from the returned directory.
The error is in the TS runtime Codegen directory returned, I need to understand now

[indigo sinew]

@edgy hollow This should fix the issue, it matched my theory
https://github.com/dagger/dagger/pull/7551 I'm waiting for the CI to be happy and we should be good

[indigo sinew]

@glad kraken You should be good to approve this one https://github.com/dagger/dagger/pull/7551
I added a test 🚀

[indigo sinew]

@half violet After talking with @kindred umbra, I decided to remove the --production flag, this would fix every issues related to dependencies downloading.
I might go back on this decision later if this impact performances to much but it shouldn't

[kindred umbra]

One of the issues about measuring the 'performance impact' is that it relies entirely on how the user has their repo set up, which you don't have any control over

[indigo sinew]

True

[pearl forge]

Just FYI for any TS module users, if you see an error containing Transforming JavaScript decorators, there is a workaround needed until our engine release next week, described here: https://github.com/dagger/dagger/issues/7583

[indigo sinew]

@pearl forge It seems the issue has been fixed on latest version tsx v4.13.3: https://github.com/privatenumber/tsx/issues/582
So we should be good with the current implementation, I'm checking rn if it's good

dagger call container-echo --string-arg "hello" stdout
hello

Yup! All good!

[pearl forge]

Awesome, can you open a PR to update our pinned dep on tsx to v4.13.3 then? Also, if there's any way of pinning that dep properly rather than in that one-off npm install -g call that would be very good, just so there's only one place we need to pin deps then

I'll open a PR to remove that temporary standalone module

[indigo sinew]

We're only using it there to execute the ts file

[pearl forge]

Right but is there a way to version it in the same way that version all the other deps of the sdk? I just don't want us to have to remember to update that one single line all the time to get new versions of it

If not that's okay, but was hoping we could use the actual lockfiles somehow

[indigo sinew]

Hmm no we cannot, or we would have to pin it in the user's package.json which can lead to broader issue

The lockfile is used to install the sdk dependencies, but tsx is a global binary that we need

[pearl forge]

Okay in that case we should just update it to the latest version we know works I think

[indigo sinew]

Okay, fine by me

It's just the execution engine, I don't think it make sense to update it on every version, just minor & major probably

https://github.com/dagger/dagger/pull/7601

@pearl forge ^^

[indigo sinew]

@glad kraken Ok my tests on enums passes but something else is failing https://github.com/dagger/dagger/pull/7498#issuecomment-2159227705
I'm not sure where this come from :/

[glad kraken]

hm that doesn't look like a flake i've seen before huh

unfortunately, i might have to leave you hanging today, i've got quite a bit of release stuff to go through 😢

[indigo sinew]

Yeah that’s fine! I can reproduce it locally

[kindred umbra]

You can just run the binary through npm / yarn without installing it globally

The sdk also has its own lockfile ... or are you discussing for the individual modules ?

[indigo sinew]

Hmm no we cannot use the lock file because tsx needs to be installed globally, not at the SDK scope

[indigo sinew]

https://github.com/dagger/dagger/pull/7636 has several breaking changes in TS dependency, I'm slowly fixing everything

[indigo sinew]

@glad kraken We got a problem, Bun doesn't implement the abort syscall used by the new execca lib:
https://github.com/oven-sh/bun/issues/11716

I can reproduce the issue locally with the latest bun version

yarn test:bun
yarn run v1.22.22
$ bun run --bun mocha

 Exception during run: 1 | export default "";
    ^
SyntaxError: Export named 'aborted' not found in module 'util'.
      at <parse> (:1:1)

error: "mocha" exited with code 1
error Command failed with exit code 1.
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

Typescript worked after couple of fixes du to the breaking changes

I'm not sure how am I supposed to take care of this, should we delay the PR until there's a fix?

[glad kraken]

hm bleh

that's kinda annoying

let's downgrade execa to the highest version that doesn't have that

solved 👀

[indigo sinew]

I spent 1 hour finding out how execca new API worked

Guess I don't have the choice for now though

[glad kraken]

yeah sorry 😢

let's watch the bun issue in the meantime

it'll make it in at some point 🎉

[indigo sinew]

LGTM, I'm rolling back these changes

[indigo sinew]

@glad kraken Could I get an approval on https://github.com/dagger/dagger/pull/7601 please 😄

Thanks

[indigo sinew]

@solar shuttle @glad kraken Hey, could you review this PR when you have 10 minutes please: https://github.com/dagger/dagger/pull/7701 😄

[solar shuttle]

@indigo sinew, heads up on https://github.com/dagger/dagger/pull/7498, I'm going to push a change. In case you have pending changes too.

Done

[indigo sinew]

Thanks! I already did the fix you were asking on TS

[solar shuttle]

Does this error mean anything to you?

Error: get module name: invalid character 'i' in literal false (expecting 'l')

nevermind 🙂

[kindred umbra]

I just ran into this as well. Re-running the job seemed to have gotten around it though

[quiet knoll]

ran into that yesterday when I was trying to set the default value of a function param using a class variable. Got the impression that it was because of "JSON-representability" for GraphQL.

[kindred umbra]

I'm running into this while running locally (0.11.9)

Error: Cannot find package '/src/dagger/sdk/node_modules/typescript/package.json' imported from /src/dagger/sdk/introspector/scanner/scan.ts
Did you mean to import typescript/lib/typescript.js?
    at legacyMainResolve (node:internal/modules/esm/resolve:214:26)
    at packageResolve (node:internal/modules/esm/resolve:840:14)
    at moduleResolve (node:internal/modules/esm/resolve:910:20)
    at defaultResolve (node:internal/modules/esm/resolve:1130:11)
    at nextResolve (node:internal/modules/esm/hooks:749:28)
    at resolve (file:///usr/local/lib/node_modules/tsx/dist/esm/index.mjs?1719424304678:2:3893)
    at nextResolve (node:internal/modules/esm/hooks:749:28)
    at Hooks.resolve (node:internal/modules/esm/hooks:237:30)
    at handleMessage (node:internal/modules/esm/worker:199:24)
    at Immediate.checkForMessages [as _onImmediate] (node:internal/modules/esm/worker:141:28) {
  code: 'ERR_MODULE_NOT_FOUND'

It looks like npm is installing typescript into the root node_modules instead of the one in the sdk

My guess is that its not loading in the pnp loader

so node is looking in node_modules for everything, but they won't be there

I ran yarn install and it seemed to fix it. My module is still currently using npm though

[indigo sinew]

Yeah it's always using yarn install, I'll work on the runtime this week to figure out which package manager it's using and changed it in the installation process

[kindred umbra]

I'm running into this while running

[long marlin]

When I run the hello-dagger example locally, I get an error on the yarn install step of Module.initialize:

error glob@10.4.4: The engine "node" is incompatible with this module. Expected version "14 >=14.21 || 16 >=16.20 || 18 || 20 || >=22". Got "21.3.0"
error Found incompatible module.

3 hours ago, glob released version 10.4.4, which changed the engine requirement from >=18 to this. Is there an unpinned dependency somewhere in dagger that is causing things to break?

[white prawn]

I'm seeing the same error in our pipelines. I've skimmed recent changes but I don't see anything obvious from recent changes that could case this: https://github.com/dagger/dagger/compare/540d884cf4583313356ebcb21b97da2b926122d1..HEAD

Looks like the node-globs package does not support node 21 from it's recent updates:

https://github.com/isaacs/node-glob/commit/d06c8f8c8288c89a4892f4ebcc23ca21840aa4a1

https://github.com/isaacs/node-glob/commit/57226aa85e7e68dc0468ff0483832ae0478db7de

Not sure if this is the right file, to fix would we just need to pin the version for node-glob here in the TypeScript SDKs package.json? https://github.com/dagger/dagger/blob/main/sdk/typescript/package.json

[indigo sinew]

Yeah I'm currently checking if that could fix the issue

The dependency is pinned in the module yarn.lock, but it seems it doesn't use it because npm, there's several thing I could do to fix that but first to fix the issue rn is pining the dep

Another way would be to move node to lts (20)

Because it seems it's not only glob, I can see the error with lru-cache too

[white prawn]

Know you've mentioned there's a few fixes, would moving to Node 22 be one as well? Not sure if that would break other things though

[indigo sinew]

Yeah but Node 22 isn't lts for now sadly

That was my original plan but we close the PR because of this

Moving the runtime to node 20 fixes the issue, I'm opening a PR

Hopefully it's not breaking anything, if anyone wanna try the PR on its project, that would help 😄
cc @long marlin @white prawn

FYI @solar shuttle @glad kraken @half violet

[indigo sinew]

https://github.com/dagger/dagger/pull/7850 PR's open

I'm waiting for the CI, writing a changelog and we're good for the merge

I'm also gonna prioritize the work on configuring the packageManager & the node version from the package.json, so we don't hit this issue anymore

[glad kraken]

wait, what's going on here? i thought we'd switched typescript sdk to using yarn everywhere?

what's the impact of this? is this impacting all pipelines written in ts? is there a fix that doesn't require updating dagger?

[indigo sinew]

Yes, but some people are installing their deps with npm or pnpm, which resolves deps differently and sometime something works with npm but not yarn, which can break things

[glad kraken]

trying to gauge if we need to release a v0.11.10 today

[indigo sinew]

What happens here is that some dependencies dropped the support for Node 21, and it broke the TS runtime

So what I did is downgrading the Node version in the runtime so everything's will be okay

And, to prevent from new issue, I'll add 2 fields in the package.json so the user can config the node version & the package manager to use, so if there's any issue because of the pkg manager, the user can change it to the one he needs

[glad kraken]

this sounds good to me

[indigo sinew]

PR merged

[glad kraken]

in the meantime, what do users have to do? i think it's okay until v0.12 to suggest to users that they should use yarn to manage and install their typescript deps

[indigo sinew]

The user will need to use the latest dagger version that use Node LTS in the runtime

so cutting a v0.11.10 release and upgrading dagger basically

[glad kraken]

not perfect, but the alternatives are:

• ship the downgrade to 20 pr immediately, which may/may not be a breaking change, which i very much want to not do in v0.11.x
• ship the incoming 2 fields package.json pr immediately, but this is stressful and i'm not convinced we should rush this change, but we may have to

also let's 🧵

[white prawn]

Happy to give this a go @indigo sinew , would the git ref be the best way to test the SDK?

[indigo sinew]

Hmm because the SDK pull the sources from the dagger binary, it might be better to clone the repo,then ./hack/dev in the dagger repo and set a .envrc in the directory you wanna try

DAGGER_SRC_ROOT=<path to dagger repo>
export _EXPERIMENTAL_DAGGER_CLI_BIN=$DAGGER_SRC_ROOT/bin/dagger
export _EXPERIMENTAL_DAGGER_RUNNER_HOST=docker-container://dagger-engine.dev
export PATH=$DAGGER_SRC_ROOT/bin:$PATH

But first try with the git ref

I know we did some change on that

@glad kraken Maybe you can give some hints?

[white prawn]

Ah right makes sense, though I'm thinking more about shipping a fix for our actions that are failing

Let me give the git ref a spin quickly here

[indigo sinew]

I'm going to do the customization of the node version from the package.json, that would fix your issue too, we may cut a release with that today, I'm doing my best to get that done quick

[white prawn]

You're killing it Tom thank you for everything you've done to help fix this, apologies don't mean to add any pressure!!

[indigo sinew]

Thanks! 🚀

Let me know if pinning the ref fixes your issue btw

[white prawn]

Hmm getting this error:
✘ ModuleSource.resolveFromCaller: ModuleSource! 0.0s ! failed to collect local module source deps: failed to get git module sdk: failed to load sdk module github.com/dagger/dagger/sdk/typescript/standalone@ee91be20b8d9d9314e88d7a6f4c504fec51d332c: select: module name and SDK must be set

[indigo sinew]

Yeah I guess you'll need to do the local setup, I've opened https://github.com/dagger/dagger/pull/7852 that should fix your issue in a different way

[white prawn]

Nice! So once shipped I can define "name": "node@lts", in the dagger.json is that correct?

[indigo sinew]

It's a field in the package.json:

{
  "dagger": {
    "runtime": "node@lts"
  }
}

You can already use it to configure bun or node, I added extra logic to also considered a version that will override the runtime default's

I also plan to add "packageManger" to configure the packageManager, I'm waiting for this one to be merged before doing

By doing so, you can fully configure your dagger TS runtime environment, which would reduce the scope of issues due to env diff between your local project and your CI

[white prawn]

Nice that'll be a really excellent fix thank you @indigo sinew !

[indigo sinew]

Thanks! Hope it will solve all deps issues people meet, Node is soo annoying with his deps system...

[white prawn]

Haha yeah definitely a pain in my experience, though these improvements feel like a solid improvement so we can better solve version pins

[white prawn]

Just so I know what to communicate internally to our teams, would it be safe to say that the above will be in a hot fix today?

[glad kraken]

maybe, i'd hold off on that for now

i'd be happy to ship the current fix, but the package manager change is really the important one imo here right?

[indigo sinew]

No, it's the version pinning

[glad kraken]

that's the root cause of the issue (pinning the node version just helps mitigates the symptoms, right?)

the issue is that a dep update, with new version constraints - we can allow users to specify those node version constraints, but the dep pinning is the real issue, no?

maybe i'm misunderstanding

[white prawn]

I think you got it. Because it's using npm and not yarn, we can ship a yarn.lock with the pipeline together with pointing to yarn in the package.json config, it should pick up the pinned versions

I think that's why Tom's looking at the packagemanger update

[indigo sinew]

What happened is:

Our TS runtime use Node 21, but the support has drop yesterday based on https://nodejs.org/en/about/previous-releases
To fix that, I did 2 things:

• Set the default Node runtime to LTS in https://github.com/dagger/dagger/pull/7850
• Allow customizing the runtime version from the package.json: https://github.com/dagger/dagger/pull/7852

The other issue is that yarn isn't using the pinned yarn.lock inside the sdk directory.
To fix that, I want to allow customizing the packageManager but it might not solve the pinning deps version issue...

What I'm wondering is, do yarn consider the yarn.lock in the sdk dir? When it install the deps?

[glad kraken]

agreed, but the issue isn't that support dropped yesterday, the issue is that a dep updated it's package.json to not include support for node 21 - if we had this dep pinned, the issue wouldn't have appeared

[indigo sinew]

Yeah, and I think the pin isn't working properly

[glad kraken]

yeah okay, cool

what i'm saying is i don't think we can ship a patch release until we understand what's happening with the pinning

[indigo sinew]

Ok I think I understand

In the runtime, I'm doing 2 yarn install

1 -> In the SDK directory which has a yarn.lock, so yarn use this lock file
2 -> In the user directory (parent dir), this one has no yarn.lock and because sdk/package.json do not pin these deps inside its package.json, it's not installing the pinned version

I think it makes sense

So 2 solutions:
1 - Checking if it's possible to specify an external lockfile
2 - Adding the sdk yarn.lock to the user's directory but it wouldn't consider external user's deps so we need to be cautious with that

I'm going to try some stuff

Ohhh okay now I see one of the issue, I need to let the yarn.lock having freezed dependencies and not pinned with ^, because it will not install the actual version but instead the latest minor of this

https://maxleiter.com/blog/pin-dependencies

I'll see how I can do that

[indigo sinew]

It seems that if you copy the yarn.lock from the sdk directory inside your source code dir, it fixes the issues and take the yarn.lock into account

Tested with dagger v0.11.9

Do you think you could try that in your CI @white prawn

[white prawn]

Trying now one sec

Like this? Seeing the same issue when run locally

[indigo sinew]

No, in the parent dir

in your node dir

So the runtime can catch it

[white prawn]

Got the same error that way unfortunately

[indigo sinew]

Can you check what's the content of lru-cache in your lockfile?

I have

lru-cache@^10.2.0:
  version "10.2.2"
  resolved "https://registry.yarnpkg.com/lru-cache/-/lru-cache-10.2.2.tgz#48206bc114c1252940c41b25b41af5b545aca878"
  integrity sha512-9hp3Vp2/hFQUiIwKo8XCeFVnrg8Pk3TYNPIR7tJADKi5YfcF7vEaK7avFHTlSy3kOKYaJQaalfEo6YuXdceBOQ==

[white prawn]

lru-cache@^10.2.0:

[indigo sinew]

And the version?

[white prawn]

Let me dump it and retry

lru-cache@^10.2.0:
  version "10.4.2"
  resolved "https://registry.yarnpkg.com/lru-cache/-/lru-cache-10.4.2.tgz#78c38f194b747174cff90e60afabcae40c3619f2"
  integrity sha512-voV4dDrdVZVNz84n39LFKDaRzfwhdzJ7akpyXfTMxCgRUp07U3lcJUXRlhTKP17rgt09sUzLi5iCitpEAr+6ug==

[indigo sinew]

Yeah that's the issue

See the diff in version?

Where does your lockfile come from?

[white prawn]

I've generated it from our package.json

Can send it to you privately one sec

[indigo sinew]

Yeah so it install already updated versions...

Hmm

[white prawn]

I could pin that package to 10.2.2

[indigo sinew]

Could you try using the lockfile from the sdk dir in dagger?

The one in our repo I mean

Should have the good pinned version

This is the one I used

[white prawn]

It's running - just cross-referencing where that's coming form in the SDK so that my package.json has the same dev dep. I think it's mocha

[solar shuttle]

Python uses a lock file, this should be similar. Can you summarize what the issue is? I'll comment next on how it works.

[indigo sinew]

The lockfile of the TS SDK isn't the same as the one generated when installing user's dep

Which create issues when the versions are differents

To workaround that, I gave him the lockfile of the sdk dir, to put in his module dir

So the runtime can use it

[solar shuttle]

In Python:

• During dagger init create a new lock file for the project and add it to the generated sources for the user. This should include the SDK's dependencies.
• During dagger develop and dagger call (i.e., non init), use the lock file from the module's sources to install the dependencies.

[indigo sinew]

Yeah, I'm not installing the lockfile during dagger init

[solar shuttle]

You don't use the SDK's lock file. You create a new one for the user and use theirs.

If their lock file doesn't change, it should get the same dependencies.

[indigo sinew]

It doesn't, that's the thing

yarn install ./sdk creates a new lockfile with different version that the one existing in sdk/

Because it takes the latest minor version

[solar shuttle]

That's ok. On dagger init you shouldn't have the user's deps pinned to what the SDK is pinned. It should be like a library. The SDK's pinned dependencies are for development and CI. But should be left open when used as a dependency.

[white prawn]

You've probably already considered this Tom but to be thorough, could the internals just dep on the user's lock file if it exists?

[solar shuttle]

That's what needs to happen.

[white prawn]

Then fallback to the SDK version

[solar shuttle]

Basically, a new dagger init should get a lock file with latest versions, constrained by the SDK's package.json (not the SDK's lock).

And on subsequent runs, the runtime should only install from the users lock in the module. Not any lock in the SDK.

[indigo sinew]

Yeah I know, but for deps of deps, it's not pinning them correctly

Like lru-cache

I'm not pinning this deps in my package.json

So it installs it with a lossy version

lru-cache@^10.2.0:
  version "10.4.2

But in my sdk lock file, it's 10.2.2

Do you see the issue?

[solar shuttle]

Two things:

• Does yarn/npm have an option that help you pin it correctly?
• If not, you can always constrain in the SDK's package.json to fix cases like this.

[indigo sinew]

Yeah, but I cannot pin every sub package version, what if there's 20 broken deps because of that?

[solar shuttle]

No I don't see it yet, because I'm not entirely sure how you're getting to that result.

[indigo sinew]

dagger init --sdk=typescript

yarn install

[solar shuttle]

Lock file should pin every dependency, even dependency of dependency.

[indigo sinew]

And you'll see that yarn install creates a lockfile with different version of deps, than the one in sdk

I know, but the user's module, isn't considering the sdk lockfile

[solar shuttle]

I've already explained, you don't want to create a lock file that's the same as in sdk.

[solar shuttle]

It shouldn't. I already explained.

Want to sync?

[indigo sinew]

If it shouldn't that's fine, but that means it fails on that kind of issues, so as you said, I should pin every flacky deps in my package.json.?

Yeah we can sync

[solar shuttle]

Dev audio?

[indigo sinew]

Yeah

[white prawn]

I can't get my audio to work but I'll listen in if that's ok

[kindred umbra]

What I'm wondering is, do yarn consider the yarn.lock in the sdk dir? When it install the deps?
The issue is that the 'root' app will install the dependencies and not look at the yarn lock file in the sdk

All dependencies should also be pinned.

[indigo sinew]

Ok I found a way to add constraint over deps of deps

@solar shuttle It's possible to add a resolutions fields, that will fix the lock file

[kindred umbra]

yarn set resolution will work

[indigo sinew]

{
  "dependencies": {
    "@dagger.io/dagger": "./sdk",
    "typescript": "^5.3.2"
  },
  "resolutions": {
    "lru-cache": "<10.4.0"
  }
} 

Yeah

@white prawn By doing so, I think it's possible to override the broken version without waiting for a new release nor updating a lockfile

It fixed the issue with dagger v0.11.9, without any other updates

@solar shuttle I'm going to check for the lockfile thing, Harley's issue should be fixed now though

[white prawn]

Testing now!

[kindred umbra]

I'm not able to get the sdk installed since its trying to do a yarn install "internally" (on the sdk)

[white prawn]

That seems to be working now!

[white prawn]

Can confirm fixed in our pipelines used across our codebase as well, thanks for the hand guys!

[kindred umbra]

From a blank slate (deleted the sdk and node_modules) it still fails to install the sdk for me

Error: failed to get module SDK: input: moduleSource.withContextDirectory.asModule resolve: failed to create module: select: failed to update module dependencies: failed to initialize dependency modules: failed to initialize dependency module: select: failed to initialize module: failed to call module "node" to get functions: call constructor: process "yarn install" did not complete successfully: exit code: 1

Stdout:
yarn install v1.22.19
info No lockfile found.
[1/4] Resolving packages...
[2/4] Fetching packages...
info Visit https://yarnpkg.com/en/docs/cli/install for documentation about this command.
Stderr:
warning package.json: No license field
warning package.json: "dependencies" has dependency "@dagger.io/dagger" with range "./sdk" that collides with a dependency in "devDependencies" of the same name with version "file:sdk"
warning ../package.json: No license field
warning package-lock.json found. Your project contains lock files generated by tools other than Yarn. It is advised not to mix package managers in order to avoid resolution inconsistencies caused by unsynchronized lock files. To clear this warning, remove package-lock.json.
warning No license field
warning "dependencies" has dependency "@dagger.io/dagger" with range "./sdk" that collides with a dependency in "devDependencies" of the same name with version "file:sdk"
(node:7) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
(Use `node --trace-deprecation ...` to show where the warning was created)
error lru-cache@10.4.2: The engine "node" is incompatible with this module. Expected version "14 || 16 || 18 || 20 || >=22". Got "21.3.0"
error Found incompatible module.

Its odd that it says its using v1.22.19 of yarn and that there is no lock file.
I also don't have @dagger.io/dagger listed as a devDependency

[indigo sinew]

Do you have it listed in dependencies?

[kindred umbra]

ya I do

[indigo sinew]

What's your current package.json?

[indigo sinew]

@solar shuttle I made a PR to detect the used package manager & its version and use it.

https://github.com/dagger/dagger/pull/7864 Still wip, specially on the test side but it's progressing.
This will also generate a lockfile for the user module, yarn v1.22.22 by default.

It seems yarn v4 breaks dagger, still investigating

/cc @kindred umbra I'm pretty sure you'll like this one ^

[half condor]

neat TypeScript example 📣 https://pod.gerhard.io/6

[indigo sinew]

@solar shuttle PR ready with the package manager & lock file generation https://github.com/dagger/dagger/pull/7864

[indigo sinew]

I upgraded the TS SDK to use no-experimental decorators @solar shuttle https://github.com/dagger/dagger/pull/7944
This fixes https://github.com/dagger/dagger/issues/7499

[indigo sinew]

@solar shuttle Ready for another review 😄
https://github.com/dagger/dagger/pull/7864

[solar shuttle]

@indigo sinew, quick review for https://github.com/dagger/dagger/pull/7967 ?

[indigo sinew]

@solar shuttle Hey, could I get your thought on https://github.com/dagger/dagger/pull/7864#issuecomment-2250508180
It's the last thing to take care of before merging it I think

[ashen imp]

Hello all, sorry if this was posted before, I didn't find any solution yet when searching in here/Github.
I want to start implementing Dagger and I've tried a couple of times - granted it was for 30 minutes or so but always the bottleneck I bumped into is node_modules.
Since (IIUC) Dagger is using a proces similar to 'docker build' with each command, copying node_modules takes forever in terms of build time. In the CI it's solvable by using a remote cache maybe that will be cached directly into the container, but what about local environment?
Anyone managed to resolve it with minimal interruption to the rest of the team?

[bright mountain]

@ashen imp typically you would use cache volumes to handle node modules and other tool-specific caches: https://docs.dagger.io/manuals/developer/cache-volumes

[bright mountain]

https://docs.dagger.io/cookbook worth looking for examples here

[ashen imp]

Thanks @bright mountain , I'll give it a go

[white prawn]

Is there a way to set a class member to an object or record? For example consider this example: https://docs.dagger.io/manuals/developer/state-functions and setting the "name" member to name: object = {}, this throws Error: Unsupported type {}. Use case here is to save collect metrics from each function in the module so I could for example get the duration of the "build" function by calling this.name.build. Is there a way to do this?

[indigo sinew]

Hey, I just saw your question sorry!

You would need to create a real type instead of using object, or make it internal so it's not caught by the introspector

[indigo sinew]

@half violet Hey, I pushed a PR to update TS dependencies, let me know when you have a moment to review it 😄
https://github.com/dagger/dagger/pull/8113

I still cannot update execca, but will happens soon when bun release its new version, so I can keep it for a follow up 🚀

[kindred umbra]

Looks like node is removing corepack
https://socket.dev/blog/node-js-takes-steps-towards-removing-corepack

[indigo sinew]

I'll need to update the runtime once we end up using this new version then 😦

[hexed tree]

What's the status of Bun or Deno without NodeJS? Is this still broken?

[indigo sinew]

You can use the Bun runtime with the TypeScript SDK module
You can find some docs here: https://docs.dagger.io/manuals/developer/typescript/356352/runtime#bun

It's also possible to configure the runtime version itself, adding @xxx to the runtime.

I'll work on a doc this week!

[finite fox]

trying to use the interactive .terminal() feature and i'm getting this:

Error: build.terminal(...).withExec is not a function

I tried upgrading to the latest dagger (0.11 => 0.12.5, which was a little challenging w/ my dagger functions tbh) but I'm still getting the error

[glad kraken]

heya 👋 this is a breaking change introduced in https://github.com/dagger/dagger/releases/tag/v0.12.0

you can see the change in the changelog, which points to https://github.com/dagger/dagger/pull/7586, which explains the logic behind the breaking change, etc

you'll also need to update your dagger.json engineVersion field, which you can do automatically with the dagger develop command

[finite fox]

ah, im actually expecting a container, but im not getting one; the dagger engineVersion looks correct (0.12.5)

it seems like its still running on 11?

ill try dagger develop

didn't know about that. wish i did lol

[finite fox]

thanks, that helped. i had nested dagger.jsons that were causing an issue; the root one wasn't updated correctly on my end.

i basically have a module w/ several submodules and missed one

[nocturne basin]

When is deno coming? really don't want to deal with ts-node/esm and ".mts" in 2024

[indigo sinew]

We do not plan to support Deno for now but there's a module runtime available that should allow you to use Deno as SDK: https://github.com/fluentci-io/daggerverse/tree/5854a23ff9e8c8147aed665874425afcec9176e0/deno-sdk
https://daggerverse.dev/mod/github.com/fluentci-io/daggerverse/deno-sdk@5854a23ff9e8c8147aed665874425afcec9176e0

You can try to use it with dagger init --sdk=github.com/fluentci-io/daggerverse/deno-sdk@5854a23ff9e8c8147aed665874425afcec9176e0 --name=xxx

Latest update is 2 months ago, it should support v0.11.7

[nocturne basin]

i only fixed the graphql-tag import and that made dagger successfully run on deno

https://github.com/dagger/dagger/issues/4368#issuecomment-2321488807

maybe you can reconsider? unlike bun, deno is not a joke 😄

at least i got the hello world typescript pipeline to work

[indigo sinew]

We plan to support it no worry, but not for now, it requires a bigger amount of work than Bun because Deno isn't fully Node compatible AFAIK, so it requires us to convert some parts of the code to be compatible on Deno.

If one day Deno becomes 100% compatible with TypeScript, it will be a piece of cake to support it

[nocturne basin]

i know that bun claimed to be a "drop-in" replacement but that was not really true and the hype quickly died when ppl found out.

deno is working to be fully compatible with 2.0 and but its already so close.
i have really good experience already using a lot of libs via the "npm" identifier. There are some ESM issues (like with graphql-tag it seems), but many things just work™️

i may dig a little into the dagger code, though i am not a go developer at all

[indigo sinew]

Good news, the TS SDK is in TypeScript 😉
If you want to contribute to Dagger and help us supporting Deno, we can discuss it in a call 😄

Last time I checked, the hardest part was converting the imports to Deno's import, I'm not sure it's possible to simply do import xx from xxx.ts, that's the big bottleneck for us because we do not want to maintains both Typescript (Node/Bun) and Typescript Deno codebase, it's twice the effort :/

[nocturne basin]

i think the biggest pain with imports is the CJS to ESM transition in the community. Plenty of tools in node just can't handle it.
(we threw out jest and replaced it with vitest simply because of ESM issues)

since deno aims to be fully compatible, i think there is a good chance to make it work tranparently

[indigo sinew]

That would be amazing

[nocturne basin]

deno doesnt demand the extension when you import cjs or "npm:" modules

and the code doesn't need the "npm:" prefix either, that can be handled in a importMap

[indigo sinew]

What we don't want is to support 2 different codebase for the TS SDK, if there's a way to configure Deno to use Node code, we'll be happy to support it

[nocturne basin]

do you have a test suite for the ts-sdk? i only found https://github.com/dagger/dagger/tree/main/sdk/typescript/test is there something you have for checking bun?

[indigo sinew]

Yes there's more!

First we run both test suit in their specific runtime in our ci: https://github.com/dagger/dagger/blob/96cb64613ff911cb93f3758423f52983b0f70955/.dagger/sdk_typescript.go#L114

Then we have specific TypeScript tests to verify the runtime configuration:
https://github.com/dagger/dagger/blob/907c013a020eb9801af7166a84e5076719efb361/core/integration/module_typescript_test.go#L506

Because Bun & Node are fully compatible, we don't tests everything in the 2 runtime, because the code run is the same in the 2 cases.

The TS suite is taking care of the internal TS SDK tests, other's tests the Dagger logic 😄

[nocturne basin]

i asked about the module resolution in deno community and a maintainer instantly reported this as a bug in deno https://github.com/denoland/deno/issues/25311 🥳

i love solving stuff by just talking to people

[indigo sinew]

Amazing!

[hexed tree]

I'm assuming we can't use generics in our functions?

@object()
class NodeJS {
  @func()
  withPackageManager<T extends PackageManager>(
    type: new (directory: Directory, version: string) => T,
    directory: Directory,
    version: string
  ): T {
    return new type(directory, version);
  }
}

?

It also seems that @func() on methods on an abstract class aren't detected as functions

[hexed tree]

Also, when using the runtime: bun, I'm still getting yarn problems:

Stdout:
Installing yarn@1.22.22 in the project...
Internal Error: Error when performing the request to https://registry.yarnpkg.com/yarn/-/yarn-1.22.22.tgz; for troubleshooting help, see https://github.com/nodejs/corepack#troubleshooting

Does anyone know how I get Dagger to stop using corepack to install yarn?

[nocturne basin]

https://socket.dev/blog/node-js-takes-steps-towards-removing-corepack fyi 😐

[hexed tree]

Sorry, I meant to stop is using that

I've got the runtime set to Bun and it keeps trying to use yarn

It's getting rather frustrating

[hexed tree]

^^ This was from the init command injecting a yarn 1.22 packageManager JSON property 😿

[hexed tree]

Hey @indigo sinew , would it be possible for https://github.com/dagger/dagger/blob/main/sdk/typescript/introspector/scanner/abtractions/typeToTypedef.ts to support constructors and functions, like:type newPackageManager<T> = new (directory: Directory, version: string) => T; ?

It also seems that @func() on methods

[indigo sinew]

Hey, could you write an issue with your needs? And I'll do my best to make that happen!
There are some technical limitation but I'll dig and find a way

[nocturne basin]

is the old connect((client) => {}) api going to be removed?

[bright mountain]

No plans to remove it, but we are moving away from it as a default, to harmonize client code around the dag. syntax

This helps with documentation, if the same code snippets work whether you're calling the Dagger API from an external client, or from a Dagger Function

[nocturne basin]

ok thanks

i am on and off checking on dagger to see how it matures and the latest TS-SDK update really confuses me

[rare hollow]

how to build from source? lil bit confused by workflows ...... i want to try with some custom image ref in sdk/typescript/runtime/main.go for ts module

[bright mountain]

@rare hollow dagger call --help to see available functions. Unsurprisingly, we use dagger pipelines for our builds 🙂

dagger call dev terminal will give you an interactive terminal with a working local build of dagger

From a PR: dagger call -m github.com/dagger/dagger/pull/$PR/merge dev terminal

From main (no local checkout necessary): dagger call -m github.com/dagger/dagger dev terminal

[rare hollow]

dagger way of doing things 🙂 unlike actions

[bright mountain]

You can also get a CLI build for the arch of your choice: dagger call -m github.com/dagger/dagger cli binary -o ./dagger-dev

Run a docs server: dagger call docs server as-service up

Run the SDK tests: dagger call sdk typescript test

For SDK lint + test + test-publish: dagger call check --targets=sdk/typescript

[kindred umbra]

The typings for dag.withNewFile seem to be incorrect (0.12.4)
The typings show as

(path: string, contents: string, opts?: ContainerWithNewFileOpts): Container

However if I change to usage to follow that, I get this error

error: parse selections: parse field "withNewFile": init arg "contents" value as dagql.String (String!) using dagql.String: cannot create String from map[string]interface {}

[hexed tree]

Slowly came to the realisation that yarn is the only possible option for Dagger and a monorepo because we need PnP and a global node_modules

I can't find a way to make Bun work like this

Anyone else found the secret codes?

[hexed tree]

I'm now worried Yarn with PnP isn't the answer. This will solve the dependency resolution problem, in that we can handle it in one step for all projects; but now the concern is this will invalidate the cache for all downstream consumers of node_modules as they don't have an isolated view if that directory

Which means I'm likely in my last ditch effort which is to get dependencies through dagger directory calls and make them local to the code; which means nobody can work locally

@bright mountain do you have any ideas here?

I assume Go Workspaces have the same challenges.

[hexed tree]

Wait, cache volumes don't invalidate pipeline cache; right? 😀

[bright mountain]

nope

[bright mountain]

they don't because Go supports global cache & concurrent access

so it all just works

the only issue is size, if you wipe it's global to all your projects

[hexed tree]

OK. Take 17 of getting this to work, wish me luck 😄

[sterile shuttle]

is there any way to create dagger module using deno? i tried to create demo using esm and npm specifier but none of them are working. getting below error

Unable to resolve signature of method decorator when called as an expression.
  Argument of type 'ClassMethodDecoratorContext<DenoDeploy, (stringArg: string) => Container> & { name: "containerEcho"; private: false; static: false; }' is not assignable to parameter of type 'string | symbol'.deno-ts(1241)

import {
  Container,
  dag,
  Directory,
  func,
  object,
} from "https://esm.sh/@dagger.io/dagger@0.12.5";

@object()
class DenoDeploy {
  /**
   * Returns a container that echoes whatever string argument is provided
   */
  @func()
  containerEcho(stringArg: string): Container {
    return dag.container().from("alpine:latest").withExec(["echo", stringArg]);
  }

  /**
   * Returns lines that match a pattern in the files of the provided Directory
   */
  @func()
  async grepDir(directoryArg: Directory, pattern: string): Promise<string> {
    return dag
      .container()
      .from("alpine:latest")
      .withMountedDirectory("/mnt", directoryArg)
      .withWorkdir("/mnt")
      .withExec(["grep", "-R", pattern, "."])
      .stdout();
  }
}

[sterile shuttle]

this is a blocker https://github.com/dagger/dagger/issues/4368 in adoption of dagger by 94K starts dev community of deno, and it's just 2 min work of bundling only...Please consider this in priority....and solve this
We are running both meetups dagger and deno at local level and so it's blocking us many times.
We don't use node anymore now for anything...

PLEASE HELP...

[sterile shuttle]

@indigo sinew Please solve this or release another package of typescript on jsr.io as ESM

[indigo sinew]

Hey, you should check this suite of message that explain why we cannot support Deno for now: #typescript message

[indigo sinew]

and it's just 2 min work of bundling only...Please consider this in priority....and solve this
How can we do such thing? What we want is a true deno SDK, is there an easy existing way to convert TS ESM to Deno?

[sterile shuttle]

deno natively supports ESM? isn't it?

[indigo sinew]

It doesn't support the ESM import import xxx from xxx, and that what we would need to run our TS SDK in a Deno runtime

[sterile shuttle]

i have seen this but we want to develop our in house dagger modules with deno

[indigo sinew]

So you should check: https://github.com/fluentci-io/daggerverse/tree/5854a23ff9e8c8147aed665874425afcec9176e0/deno-sdk from the message I linked above

It's a deno SDK runtim that you can use to create Dagger module with deno

It's not perfectly maintained though since it's from the community

[sterile shuttle]

thanks a lot forz your support but we need officially support things to maintain/run production code.

[indigo sinew]

If only Deno was 100% compatible like Bun does, we would have no issue creating a Deno runtime for the TS SDK, right now it requires duplicating the codebase to use Deno specific import syntax

[sterile shuttle]

can this help? https://deno.com/learn/modules

[indigo sinew]

This is Deno to Node, we need Node to Deno :/

[sterile shuttle]

Have you checked deno 2.0 rc features, can it solve?

[indigo sinew]

Maybe this can help: https://github.com/garronej/denoify

[sterile shuttle]

Okay then can you please solve this soon?

[indigo sinew]

I can give it a try, but it will be in experimental state for some time until it's bullet proof tested (like bun) so I'm not sure it's a good idea to use it in production for now

Is it a big blocker for you?

[sterile shuttle]

Yes from the last many months.

[indigo sinew]

You could first try the TS SDK with a small and then smoothly move to Deno once we support it

[sterile shuttle]

Ya that's fine, if we can use atlist is better

[sterile shuttle]

Ts sdk is not working with deno

[indigo sinew]

Yeah but you can create your dagger module in Node, why do you specifically need Deno?

You'll run your dagger module inside the runtime so the language doesn't really matter

[sterile shuttle]

We are running 100% all code all projects with deno only.
No node

[sterile shuttle]

Is there any example available for this to show to the team?

[indigo sinew]

I see, however note that even if you write Deno code, you'll not be able to run your module directly with deno run xxx, you'll need to use the dagger CLI so the DX would be exactly the same (Deno, or Node, or Go)

Sure, you can check our docs: https://docs.dagger.io/ and our own CI: https://github.com/dagger/dagger/tree/main/.dagger

[sterile shuttle]

so you are saying like node typescript sdk is calling dagger cli under the hood, and so similar thing we can do directly with deno code by calling dagger cli, so ts sdk won't require in this path right?

[indigo sinew]

It's the reverse, the dagger cli is calling the TypeScript runtime to run your Dagger module

[sterile shuttle]

so dagger cli will call deno to run TS code?

[indigo sinew]

Exactly

[sterile shuttle]

how it is possible?

[indigo sinew]

We built runtime module for every language, example for typescript: https://github.com/dagger/dagger/blob/main/sdk/typescript/runtime/main.go

So your module is loaded inside the runtime, then executed. Inside this runtime, your module can use the dagger SDK to perform operations inside the Dagger engine (building a container, running tests, exposing a service etc...)

So what I'm trying to explain is that the language your using doesn't matter, all your calls will be done from the dagger CLI. For example dagger call test, this will use the test function you defined inside your module.

[sterile shuttle]

ok got it then is it advisable to drop using sdk?

dagger can call functions directly without modules?

[indigo sinew]

What do you mean?

[sterile shuttle]

is this possible without using TS sdk right?

[indigo sinew]

You have to use the SDK inside the runtime, here's an example of call: https://docs.dagger.io/manuals/developer/functions/#directories-and-files

We import things from the TS SDK: import { object, func } from "@dagger.io/dagger"

[indigo sinew]

You would not be able to communicate with Dagger then, but you can use the Go SDK or whatever language you like yes

[sterile shuttle]

ok

[indigo sinew]

A dagger module is a piece of code executed in a runtime, it's not the same as your actual codebase

That's why I'm saying that you can use the Node TS SDK to create your CI, this will not conflict with your Deno codebase

[sterile shuttle]

but we can't use/write anything with node as per few projects agreements

[indigo sinew]

So you'll need to wait a bit for a Deno runtime to come up, I'll see if Denoify can do the job this week

This isn't an easy task though, but maybe it can unlock you

[sterile shuttle]

alright, thanks a lot for your support.

[indigo sinew]

If you wanna contribute, you can help me a lot by testing the work I'll do on that

[sterile shuttle]

anytime, 24x7 let me know when to start...

can't we do like releasing new package on JSR.io and it will be usable for all deno, node bun?

[indigo sinew]

No because the SDK is bundled in the dagger engine, then copied to the runtime

[sterile shuttle]

ok

is it advisable to use direct dagger graphQL api rather then sdk?

[indigo sinew]

The TS module is importing the SDK as a local dependency: @dagger.io/dagger: ./sdk

[indigo sinew]

You'll not benefit from the super cool dagger cli DX if you do that

[sterile shuttle]

ok there we will not have power of programming language, right?

[indigo sinew]

Yes exactly + no dagger cli DX so you would need to recreate your own CLI system on top of it, basically you would rollback to Dagger v0.0.2

[sterile shuttle]

haha cool...

[indigo sinew]

Just give me a couple of days and I'll try to get something, and if it's not working, we'll know why

The best thing that could happen would be Deno being able to directly run Node code

[sterile shuttle]

cool

[sterile shuttle]

yes need to wait

can't we build directly from typescript compiler?

i have created one issue for them also https://github.com/denoland/deno/issues/25669

[indigo sinew]

Hmm not sure, I also saw that: https://github.com/denoland/deno/issues/18474
meaning that we cannot import a local npm module so it will to be denoify

[sterile shuttle]

basically you need to add ts sdk in dagger runtime?
and it's needs to be locally downloaded?

deno automatically cache packages in local

[indigo sinew]

Yeah, you need to be able to import the local SDK source inside your module

try to run dagger init --sdk=typescript --name=test then dagger develo, you'll see a sdk dir is created in your current dir

[sterile shuttle]

can this help? https://docs.deno.com/runtime/reference/cli/cache/#cache-location

[indigo sinew]

After the module is denoify it might be

[sterile shuttle]

ok then waiting only now to try and giving you feedback.

[indigo sinew]

Ty

[indigo sinew]

@sterile shuttle I can see several incompatibilities whe using denoify on the TS SDK, which is not ideal at all.
Is there a way from Deno to import a local npm module? Maybe I don't need to use denoify but instead serve the npm module as a local dependency

[sterile shuttle]

this may help https://docs.deno.com/examples/npm/

[indigo sinew]

This issue: https://github.com/denoland/deno/issues/18474 is still active and not solved yet, we would need to make a Deno SDK, this would allow use to simply copy the TS SDK folder as a Node module, and then use Deno to import, then I can create a custom Deno entrypoint (https://github.com/dagger/dagger/blob/main/sdk/typescript/runtime/bin/__dagger.entrypoint.ts#L0-L1) or even execute that one with deno run

This is using an external npm package, I need to import a local one

Because the SDK is bundled in the engine

[sterile shuttle]

native way would be better for production reliability

[sterile shuttle]

yes and you can use this same file for deno sdk also https://github.com/dagger/dagger/blob/main/sdk/typescript/api/client.gen.ts
right?

[indigo sinew]

The client.gen.ts file is a generated file, but the SDK is the whole dir, needed to introspect the user's module, provide telemetry, execute operations etc...

[sterile shuttle]

hmm ok

[indigo sinew]

Maintaining both Deno & Node would be almost impossible, I'm already under the water between TS & core features

That's why I want to find a way to use the current codebase, but in Deno

[sterile shuttle]

okay got it

does sdks has all same features as same as graphql api?

[indigo sinew]

Yes, SDKs just call the gql API basically

[sterile shuttle]

yes but all features of graphql api are supported in sdks?

[indigo sinew]

But if you want, you could create your own Deno SDK, would be happy to help on that

Yes, because the SDK client is generated based on the GQL API

That's why we have a generator for every SDK

[sterile shuttle]

ok

[indigo sinew]

We got couple of community SDKs like Elixir and PHP, it's maintained by the community and works well, the only difference is that they are not officially maintained by the Dagger team but we help & review every PRs

And it's in our repo

So if you wanna build a Deno native SDK, we can help you, since it's almost the exact same as TypeScript, you wouldn't have much work to do except converting all the Node code into Deno

[sterile shuttle]

so you can help for this generation part and telemetry and other parts? and after implementing this sdk will be also integrated into dagger engine?

[indigo sinew]

Yes, as a remote module

[sterile shuttle]

ok

[indigo sinew]

The generator is in Go, but you can create a Deno generator in Deno, like Python has a generator in Python, Elixir in Elixir etc.., it's not that hard if you understand what to do

I think that's the best solution right now, having a Deno community SDK

[sterile shuttle]

ok let me confirm with the team for deno sdk contribution

[indigo sinew]

So try to motivate this community of 94K engineers to help us, and we'll be happy to make that happen

[indigo sinew]

Nice

[bright mountain]

@rotund ember is giving a demo with the Typescript SDK, and hitting an error require is not defined, does that ring a bell?

[quiet knoll]

did you solve this?

not using require() anywhere?

// Old CommonJS style
const fs = require('fs');

// New ESM style
import fs from 'fs';

[bright mountain]

yeah apparently that was it

[quiet knoll]

@indigo sinew seeing some errors with my Pulumi pipeline in TypeScript like

Error: Encountered an unknown error while requesting data via graphql

UND_ERR_HEADERS_TIMEOUT

for longer running pipelines

is the UND one related to https://github.com/nodejs/undici/ ?

[indigo sinew]

Yes, we dropped CommonJS so you're support to use import only

[indigo sinew]

Hmmm, is your engine running and accessible from the CLI?

We're using graphql-request which is a package on top of undici, so the issue is on the connection side probably, not the code itself

[rotund ember]

Thanks for the info! I should have read the changelog more closely ❤️

[rotund ember]

Hm, I am a bit confused, this example in our cookbook builds concurrently in python and go, but sequentially in typescript

https://docs.dagger.io/cookbook#perform-a-matrix-build

Is the example incorrect?

Typescript: https://dagger.cloud/levs-test-org/traces/6263761bfb5da582fff5a086984f59de

Go: https://dagger.cloud/levs-test-org/traces/d1513558be3752b202aa59146a0f1b57

Python: https://dagger.cloud/levs-test-org/traces/a5cf6da5e5821470614e5f888a981778

[celest oriole]

hey @indigo sinew -- I just got back to trying dagger. Did you abandon trying to use the latest yarn in the TypeScript SDK? 13.3 is working, but the the dagger SDK is built with yarn v1 in my monorepo whereas we're using yarn v4 locally.

[earnest tusk]

Hey folks, I just started to experiment Dagger.. Is it ok to have multiple dagger modules in monorepo with different folders?
How should I call each module or functions from these monorepo?

[indigo sinew]

Hey, you can configure your repo to use yarn v4 following that doc: https://docs.dagger.io/manuals/developer/dependencies#dependencies-on-third-party-packages

[indigo sinew]

Yup that's a good idea! This is what we do in our own CI for dagger, you can check the .dagger directory from our repo there: https://github.com/dagger/dagger/tree/main/.dagger

[celest oriole]

That's working fine for me. I was just surprised that you were using yarn V1 in the SDK after all the gyrations of that PR

[celest oriole]

The .dagger/sdk in my repo was built with yarn V1 whereas my project uses v4

[celest oriole]

Not an issue for me but want what I expected

[indigo sinew]

Is the package.json in your .dagger has packageManager set to yarn v4? Remember that's the module's code is isolated from your codebase 🙂

[indigo sinew]

Because we're still not sure which package manager we want to use by default, we're doing some benchmark to see what's the best for Dagger module 😄 so for now we adopted the most common

[celest oriole]

Gotcha -- makes sense. I pontificated on this here: https://github.com/dagger/dagger/issues/8123#issuecomment-2288607467

Anybody doing monorepos with dagger that have solved this issue? https://github.com/dagger/dagger/issues/8530

[hexed tree]

I suspect this is a bug, I'd have expected this to run concurrently

[kindred umbra]

It looks like default values aren't handled for @func

@func()
  async run(url: string = STAGING_URL): Promise<Container> { }

I have getting this error during type generation

failed to get field spec: failed to decode default value for arg "url": invalid character 'S' looking for beginning of value

[earnest tusk]

@indigo sinew but what I understood here, you have multiple functions in single file, am I wrong?

[indigo sinew]

Yes and multiples modules too called from the functions 😄

[quiet knoll]

It looks like default values aren't

[earnest tusk]

@indigo sinew actually what I meant is, consider I would like to have two different modules like docker and node..
Does it make sense to have a folder structure like this?

├── README.md
├── docker
│   ├── LICENSE
│   ├── dagger.json
│   ├── package.json
│   ├── sdk
│   ├── src
│   ├── tsconfig.json
│   └── yarn.lock
└── node
    ├── LICENSE
    ├── dagger.json
    ├── package.json
    ├── sdk
    ├── src
    ├── tsconfig.json
    └── yarn.lock

[indigo sinew]

Sure!

[earnest tusk]

Thanks @indigo sinew .. I ve another noob question.. Can we develop something on top the module in Daggerverse?
For instance, i would like to add one function to this to scan the images for instance? Can I use this module in my typescript sdk ? https://github.com/shykes/daggerverse/blob/c9a80c9eac0675a53a7e052da3594207f4235988/docker/main.go#L254

[rotund ember]

Cookbook | Dagger

[bright mountain]

Yes you can use that module frpm yours; dagger install github.com/shykes/daggerverse/docker then your dag object should have new generated bindings: dag.docker().

If you want to modify that module then you have to fork it, change the Go code etc. But depending on what functionality you want to add, you may not need to. It depends if you can implement the additional feature as a wrapper function in your own module.

how do you see that scanning feature working? would it use the docker engine specifically?

[earnest tusk]

Thanks @bright mountain

how do you see that scanning feature working? would it use the docker engine specifically?
Not specifically, I just would like to use this module as an example, and wonder if I use any module written in go with my ts module..

[bright mountain]

Yes any module can use any other module at full feature parity, regardless of language. That is a key feature of Dagger.

[earnest tusk]

Perfect! Then I can use something like below i suppose, right?

dagger. 
 () -> original function in the called module.(written in go)
 () -> new function (written via ts)

[bright mountain]

I think so, but a more concrete example would help, to make sure I understood correctly

[earnest tusk]

I meant consider below as an example..
Let's say I want to add scan() function to one of the existing modules[https://github.com/opopops/docker/blob/230c9f3121bd02566d86766fd333d65c97166454/dagger/src/main.py] in the community.
Can I use like below? Should the class name be the same as the existing module or does it matter?

@func()
example(digest: string, context: Directory, dockerfile: string, target: string): Docker {
    return dag
        .docker(digest)
        .build(context, dockerfile, target)
        .scan() -->
}

[bright mountain]

No, you can't add a function to another module on the fly like that. But you can create your own higher-level functions which "wrap" functions from another module. For example:

@func()
example(digest: string, context: Directory, dockerfile: string, target: string): Docker {
    ctr = dag
        .docker(digest)
        .build(context, dockerfile, target)
    return scan(ctr)
}

@func()
scan(ctr dagger.Container): {
  // Implement your scanning logic here
}

In the case of scanning, unless you plan on using a scanning tool that specifically depends on the docker engine API, you probably don't need my docker module because the Dagger API already has native support for 1) building from Dockerfile 2) pushing and pulling to and from a registry 3) importing/exporting a live container to/from an OCI archive 4) inspecting the filesystem of a container

@quiet knoll which is the best Trivy module to point to as an example?

I'm going to tentatively go with: https://daggerverse.dev/mod/github.com/sagikazarmark/daggerverse/trivy

@earnest tusk here's an example of scanning module you can use directly

[earnest tusk]

Actually this is what I d like to learn, and super helpful. Scanning is just for example, I definetely will check.
But the part Id like to learn is how to add a new feature to the existing module that we can install from the community.

[earnest tusk]

You meant, Dagger has already native functions ?

[bright mountain]

Yes absolutely. We call them "core functions" and "core types", the Typescript reference is here: https://docs.dagger.io/reference/typescript/modules/api_client_gen

[quiet knoll]

Seems like a good choice that is used in anger 🙂

[earnest tusk]

I think i need to understand more how this is working..
Is there any example that I can take reference except this?

[indigo sinew]

@solar shuttle @glad kraken Waiting for a review on https://github.com/dagger/dagger/pull/8544 😄

Super small PR, extra easy to review 😄

[glad kraken]

@indigo sinew was this change intentional? https://github.com/dagger/dagger/issues/8559

maybe a regression related to https://github.com/dagger/dagger/commit/f15c2996d26dbbc17bec4b6377629bdf1e934ddf ?

[indigo sinew]

Yes, no dist is generated anymore since https://github.com/dagger/dagger/commit/f15c2996d26dbbc17bec4b6377629bdf1e934ddf

But it's not necessary to execute TS code

I'll add a comment on the issue

[glad kraken]

right, but this is now a regression right?

because you can no longer call the sdk library from js code anymore

which you used to be able to do

[kindred umbra]

That seems kinda broken for non-deno usecases.
For normal esm use-cases it should be the .js files that are imported.

[indigo sinew]

But Node will support by default .ts import with the new LTS version, so why would that be an issue? 😮

[kindred umbra]

What version of node added that ?

I found this, but its behind an experimental flag
https://github.com/nodejs/node/blob/main/doc/changelogs/CHANGELOG_V22.md#experimental-typescript-support-via-strip-types

[indigo sinew]

The new LTS version will, I cannot find the link to that though

Yeah this: https://nodejs.org/en/learn/typescript/run-natively

[kindred umbra]

Thats still behind an experimental flag

[indigo sinew]

Hmm

Well I think I should revert that PR then, since deno cannot be integrated now...

[kindred umbra]

That was also added in 22.6, which has a limitation that it only strips types.
22.7 does some transpilation, but it probably still has a bunch of limitations
https://github.com/nodejs/node/blob/main/doc/changelogs/CHANGELOG_V22.md#experimental-transform-types-support

[indigo sinew]

Yeah I might have went to fast on that one, I pressured myself to integrate deno and did that change, was a mistake

Will revert it now

@glad kraken https://github.com/dagger/dagger/pull/8560 <- Will revert that

[rugged blade]

Feels like in general, it would be best to not leverage experimental node features without pretty significant vetting. Even more so for newly released experimental features.

[kindred umbra]

It also to be able to support earlier versions of node ... dont have to go back super far, but we shouldn't be expected to upgrade to latest.
There are usually reasons why people can't upgrade node to the latest all the time

[indigo sinew]

Yeah that's why I opened the revert, sorry for that!

[earnest tusk]

Hi All,
Sorry for the noise but can someone guide me how can I do this?

[indigo sinew]

Hi All,

[bright mountain]

If hypothetically I wanted to make calls to the core dagger API from a Javascript tool (not typescript), what would be simplest and cleanest way to do that?

(I'm trying to add Dagger support to an existing tool written in JS)

[rotund ember]

Are you open to using the shell to get output or do you want to do something like dagger run style SDK usage?

[bright mountain]

dagger run style ideally

I don't mind making raw graphql queries if needed. My queries are pretty straightforward without a lot of data back and forth

(I'm trying to replace ad-hoc http downloads with calls to dag.http()

[rotund ember]

I wonder if this just works? Have not tried these instructions recently but they imply that NodeJS is just supported https://archive.docs.dagger.io/0.9/sdk/nodejs/783645/get-started

[bright mountain]

We're going to start a weekly call to discuss the design and roadmap of the Typescript SDK. If you're a power user or the SDK with questions and suggestions, or if you're interested in contributing, that call will be for you 🙂

Who's interested in joining?

[sterile shuttle]

Me & @humble moon would like to join. Please let us know how we can join. Thanks

[earnest tusk]

Hey All,
I d like to start my journey with Dagger to develop some functions related Github to add issue comment ort etc with Typescript..
Does it mean can I use octokit for instance in my Dagger module?

[bright mountain]

Hello! Yes you can use any regular library in your Dagger Functions. We have a demo somewhere of a Dagger module that adds comments to a github issue, let me see if I can find it

Found it! https://discord.com/channels/707636530424053791/1214729466920706088

[earnest tusk]

This one?
https://github.com/aluzzardi/daggerverse/blob/f17b8131ea25ff620606041134a5f1b2b5a7799d/github-comment/github.go

[bright mountain]

ha ha probably 🙂

that example is in go, but you could just as easily develop it in Typescript

[earnest tusk]

My Go knowledge is too bad, but try to influence.. I have really silly and noob questions?
I thought that every function should somehow return dag.Container

When we have to use this?

[bright mountain]

Your function can return any type you want, as long as it can be defined in the Dagger API type system. Container is just one of those types.

Functions that return containers are often practical, because callers of that function can easily "chain" more functions on that container, to create custom pipelines from the result of your function without you having to think of all the extension points yourself.. But it's just a convention.

You can also create custom object types with their own functions.

[earnest tusk]

thanks @bright mountain .The examples with Typescript modules are limited, which makes it hard for me to understand how everything works.
Where can I find more examples except Daggerverse?

[humble moon]

Is it necessary to have src folder for typescript modules? if we move index.ts out from src folder then it is not detecting functions in dagger functions command

[kindred umbra]

We're going to start a weekly call to

[half condor]

<@&1113692044158836736> fans, we’re kicking off a monthly SDK call to review TypeScript SDK GitHub issues and prioritize with your input.

Join us by clicking “interested” here: https://discord.com/events/707636530424053791/1290853953776390175

Submit your issues with the “area/typescript” label to ensure they’re covered: https://github.com/dagger/dagger/issues?q=is%3Aopen+is%3Aissue+label%3Aarea%2Fsdk%2Ftypescript

[sterile shuttle]

Hey guys we need help here #1291391958668939264 message

[past grail]

deno 2.0 has been released, any news about how to use it with the sdk?

[indigo sinew]

Hey!

I didn't had a look yet, will do next week, I'm focus on other important issues this week 😄

[indigo sinew]

@solar shuttle Ready for review 😄
https://github.com/dagger/dagger/pull/8653

[bright mountain]

My big question for the Typescript SDK: should it prioritize the best DX, or flexibility? https://github.com/dagger/dagger/issues/8689

Would love to get your thoughts on this 👆 and add it to the call agenda

[half condor]

<@&1113692044158836736> users - Maintainers call starting now in dev audio - https://discord.gg/MQUDBaqN?event=1290853953776390175

[indigo sinew]

Thanks again to everyone for joining this meeting! Lot of great discussions happened and we'll do our best to integrate what we talked about on Dagger 😄

See you next month for the next one 😉

[bright mountain]

Sorry I couldn't join.. I would love to read the notes!

[sterile shuttle]

@indigo sinew any update on deno support direction?

[sterile shuttle]

https://youtu.be/7uiL4WYvZVs?si=-7r6AnbtU0jPK_nl

[glad kraken]

👋 @indigo sinew I'm starting to see a lot of ts flakes recently 🤔 https://github.com/dagger/dagger/actions/runs/11501610429/job/32014612907?pr=8765

is 2000ms too tight here?

[indigo sinew]

It shouldn't be too tight but I can extend it

[indigo sinew]

Some update related to reference by resolution:

It works for most casse but there' aresome flackiness I've found with ts-morph: https://github.com/dagger/dagger/issues/8676#issuecomment-2434996342
So I'm gonna refactor the code to stop using of TS-morph but instead just the compiler API, because it's way too flacky and gives inconsistent behaviours, I did some hacks to fix theses inconsistencies but it leaves too much room for issue sin production and these issues are not simple at all to catch.

It will take a bit more time for me to complete the feature because of that, I've been fighting the type resolution of ts-morph for almost this whole week and I think it's time to find a better solution with a cleaner type abstraction than that.

I'll keep posting updates on the issue, but I wanted to also post one here since maybe some people missed my updates

[indigo sinew]

I just opened my PR to resolve types by references: https://github.com/dagger/dagger/pull/8824

I still need to fix a few things to get all the tests to pass, but 90% of the job is done!
cc @solar shuttle sorry in advance, the review's gonna be long...

Also needs to add some changes to the doc pages to explain new supports and some integration tests

[wheat lake]

hello everyone, i try the dagger tutorial with typescript sdk but i have an error, dagger can't fetch images because he don't have credentials: Error: resolve: failed to resolve image docker.io/library/alpine:latest: failed to resolve source metadata for docker.io/library/alpine:latest: failed to get credentials: error getting credentials - err: exit status 1, out: `

I don't understand why I need to login on my dockerhub account to fetch a public images

[bright mountain]

Does it work after logging in to your docker hub account?

[wheat lake]

I think I've found the error, when I run my command without sudo, I'm not asked for my credentials, when I run the command with sudo I'm asked for my credentials even though I'm logged in, I'm running dagger with sudo because I had this error and I didn't know how to fix it. did not complete successfully: get user: unable to find user root: invalid argument

[indigo sinew]

Hmm are you able to pull the image with a simple docker command?

[wheat lake]

Yes

But now i can pull the image but i can’t run it with dagger

[indigo sinew]

That's strange! Are you using the dagger runner automatically provisionned by the CLI? It should have your docker credentials configures in by default

[indigo sinew]

If you run docker container ls, do you see the dagger runner container?

And also, what's your dagger version + operating system? Run dagger version to see it

[wheat lake]

yes the dagger container is up and running, here is my version of dagger dagger v0.13.7 (registry.dagger.io/engine:v0.13.7) darwin/amd64

[wheat lake]

I have updated my version of dagger and now it works again. Thank you for your help.

[indigo sinew]

No problem! I'm not sure why it wasn't working before

https://github.com/dagger/dagger/pull/8824 is finally merged and will be part of the next release.
Here's a copy of the release note for these who wants to try it before 😄

Add support for non-decorable keywords by resolving declarations by references.

Support native enum keyword to define enumeration without decorators.
Example

enum Example {
  A = "A",
  B = "B",
}

Support native type keyword to define type without decorators.
This can be use to define data object or primitive type.
Example

type Example = {
  a: string
  b: number
}
type Integer = number
type Foo = string

Note: If the type is a data object, all properties will be exposed
to API & CLI.

Support resolution of variable and enumerations as default values if they
are exported by the same module.
Example

export enum Example {
  A = "A",
  B = "B",
}
export const defaultValue = "foo"
@object()
class Test {
  @func()
  defaultEnumVar(value: Example = Example.A): Example {
    return value
  }
  @func
  defaultStringVar(value: string = defaultValue): string {
    return value
  }
}

[half condor]

<@&1113692044158836736> Maintainers call starting in 5 minutes on dev-audio. See you there! https://discord.com/events/707636530424053791/1296097875620794498

[kindred umbra]

My meeting invite shows it starting in an hour... seems like Daylight savings time strikes again

[half condor]

@humble moon, we'd love feedback on this doc. Thank you! https://docs.dagger.io/api/module-dependencies

[half condor]

Oh no, was the invite from Discord?

[half condor]

If you’d like, I can add you to our internal google calendar invite!

We have a few questions for the Dagger <@&1113692044158836736> users. Threads starting below...

What other Typescript syntax thing you want to be supported by the SDK?

Is there any runtime configuration that you wish you could do?

Are you annoyed by the decorators? Do you wish the syntax was more implicit?

[sterile shuttle]

Let's schedule next TS meeting before Christmas or before Dec 15 so we can achieve our goal, what you guys say

[half condor]

Right now, it is at Dec 17th, but I will see if we can update it.

[sterile shuttle]

then this is fine

[solar shuttle]

@indigo sinew, a quick search told me that if you import with .ts extension, that won't be rewritten in the final js. There's tools that you can run after transpilation to rename them for you (e.g., tsc-alias) but I think it's just simpler to remove that extension from the import. Then the transpiler should do the right thing.

[indigo sinew]

Yeah, I'll go for this solution I think

[indigo sinew]

Seems like it works by removing extension: https://github.com/dagger/dagger/issues/4368#issuecomment-2486688196

So I can do a PR that remove all .js extension, I can do it during the reordering of every imports, then do the decoupling the a follow up and then we should be good to retry a deno support

It also seems that https://docs.deno.com/runtime/reference/cli/unstable_flags/#--unstable-sloppy-imports might supports .js file but since it's very painful to see .js for .ts file, I think it will not hurt anyone to do this quick refactor/cleanup

[indigo sinew]

@sterile shuttle No need to call your team for the poc, I did it today btw (the import thing)

[sterile shuttle]

great...thanks a lot...

[indigo sinew]

@solar shuttle Hey 😄

The decoupling PR is on its way, I have not finish everything yet (I still need to fix the module execution).
It seems that because I moved the index.ts into src/, the SDK isn't able to directly execute typescript now (so I should build in the engine first, which is normal I think actually)

Still a bit of work to do, but we're close.

I also tried to remove .js extension and it completely broke mocha test because: https://www.totaltypescript.com/relative-import-paths-need-explicit-file-extensions-in-ecmascript-imports
But it seems deno 2.0 now has an option to work even with .js (the unstable-sloppy-imports)

https://github.com/dagger/dagger/pull/9031

/cc @sterile shuttle ^^^ I Did some progress today to support Deno, will continue tomorrow

[sterile shuttle]

Awesome...dagger team rocks...

Is it going to be supported in ESM manner?

[indigo sinew]

Yep!

[sterile shuttle]

Awesome.

[indigo sinew]

@solar shuttle Would be amazing to have a quick review on that on: https://github.com/dagger/dagger/pull/9031

There's a lot of changes but it's actually very simple, I'll update the PR description to give you all the details

[indigo sinew]

@glad kraken @random coyote If you have some time, could you review this little PR: https://github.com/dagger/dagger/pull/8675 😄

[wheat lake]

Hello, I'm back with an error, the error is as follows:

My version of dagger is well updated, the image I'm using is the following: ghcr.io/cirruslabs/flutter:3.7.7

[glad kraken]

Hm that error looks like it comes from failing to parse /etc/passwd

Has that file somehow been modified?

[wheat lake]

no, I don't think so. I don't know if the keystore generation action modifies this file.

i use keytool CLI

[indigo sinew]

An exciting new feature for TypeScript 😄

Follow progress on https://github.com/dagger/dagger/pull/9123

[indigo sinew]

@sterile shuttle

I finally found a way to support Deno with TS SDK, however, there are multiples issues regarding the IDE integration + overall runtime support:

• I couldn't run the TS mocha test with deno (it's important for long term support)
• The Deno IDE integration for decorators shows error while the code is actually right, I don't understand why yet (EDIT: setting the compiler option to experimental decorator actually fixes the error haha).
• deno.json must be standalone and I cannot count on the existing package.json since Deno doesn't support local dep install (https://github.com/denoland/deno/issues/18474), I could work around that by simply let the sdk dir be part of the top project but it's not good IMO.

But at least it's runnable and I could use a deno package inside a dagger module haha, we can talk about it during the next community call but I think it will not be production ready Christmas, but it can be runnable (will open a PR though if you want to give it a try)

^^^

Using deno I can list functions with dagger functions and execute denoFct with dagger call deno-fct 😄

Little trace to show the deno exec 😄

[sterile shuttle]

@indigo sinew first of all i appreciate your efforts and dagger team support also to make deno support possible in future.

And yes we can discuss in the next ts sdk meeting.

[half condor]

<@&1113692044158836736> maintainer call starting now in dev-audio https://discord.com/events/707636530424053791/1308639504533422121/1318608646963200000

[broken whale]

Issue DOCS-389 created.

DOCS-389 - document Custom Span on TS SDK

Create documentation for https://github.com/dagger/dagger/blob/eb738ebe8bf53a80f8061d377dd04934e1489fce/sdk/typescript/src/telemetry/tracer.ts#L41

Status

Triage

Assignee

Vikram Vaswani

Documentation · Dec 17, 2024, 5:06 PM

[sterile shuttle]

any github issue to track this? we would like to try this to use dagger native otel for our web app code by extending dagger inbuild otel features, please guide us if this is possible now also...even...

[charred crag]

This is in progress in https://github.com/dagger/dagger/pull/9301/

[bright mountain]

I just noticed the Typescript SDK has a Container.withRun? Did I dream that?

[sterile shuttle]

Can you please elaborate on what you want to say or ask...

[bright mountain]

I'm asking if I'm correct that the typescript SDK has a Container.withRun function builtin, and if so a follow-up question: why?

[indigo sinew]

Hmm, where did you see that? I couldn't find it on the base generated client

[glad kraken]

yeah, i can't find any instances of withRun anywhere in our codebase at all actually

[indigo sinew]

Maybe it was just a dream after all 😮

[bright mountain]

I saw it in a user's example, but I guess it was a custom Container type. That's why I was confused. Thanks!

[indigo sinew]

Typescript interface support is coming soon: https://github.com/dagger/dagger/pull/9347 😄

[glad kraken]

@indigo sinew any chance we can do something about the typescript flakes? https://github.com/dagger/dagger/actions/runs/12770326585/job/35594928990?pr=9384
could we try bumping all the test timeouts? it looks like there's a way to configure the global timeout https://github.com/mochajs/mocha/blob/main/example/config/.mocharc.yml#L47

leaving it in the current state makes it very hard to discern which failures are legitimate

[indigo sinew]

Yeah definitely!

[indigo sinew]

PR opened: https://github.com/dagger/dagger/pull/9386

[fathom obsidian]

Total noob to the TypeScript SDK (but not to TS) and sort of a noob to Dagger, so bear with me.

I had this error popping up with a fresh basically "hello world" class and method.

could not find module entrypoint: class M8a from import. Class should be exported to benefit from all features.
/src/dagger/sdk/src/module/introspector/dagger_module/module.ts:98
      throw new IntrospectionError(
            ^


IntrospectionError: could not find main object M8a in module  located at /src/dagger/src/index.ts,/src/dagger/src/
er.entrypoint.ts,/src/dagger/src/../sdk/src/api/client.gen.ts
    at new DaggerModule (/src/dagger/sdk/src/module/introspector/dagger_module/module.ts:98:13)
    at scan (/src/dagger/sdk/src/module/introspector/index.ts:18:18)
    at async connection.LogOutput (/src/dagger/sdk/src/module/entrypoint/entrypoint.ts:17:26)
    at async <anonymous> (/src/dagger/sdk/src/connect.ts:44:11)
    at async withGQLClient (/src/dagger/sdk/src/common/graphql/connect.ts:24:12)
    at async <anonymous> (/src/dagger/sdk/src/connect.ts:40:9)
    at async connection (/src/dagger/sdk/src/connect.ts:38:5)
    at async entrypoint (/src/dagger/sdk/src/module/entrypoint/entrypoint.ts:11:3) {
  cause: undefined,
  code: 'D110'
}

Node.js v22.11.0
! failed to initialize module: failed to call module "m8a" to get functions: process "tsx --no-deprecation --tsconfig /src/dagger/tsconfig.json /src/dagger/src/__dagger.entrypoint.ts" did not complete successfully: exit code: 1

And the class is called M8A. When I renamed it to M8a, the error went away.

Is this a quirk with TSX and class names? Or, an issue with the SDK?

The funny thing is too, the Dagger init process picked up that name i.e. automatically created the class with the name "M8A".

[indigo sinew]

Hey! Typescript maintainer here 👋

It's due to a case with the name of the class, I think the number in M8A confuses the registered object name 🙂
I think it's a uncovered edge cases of object name, if you can provide a repro case on a GitHub issue, I'll be happy to fix it 🚀

[fathom obsidian]

Done!

[indigo sinew]

I forget to ping you yesterday sorry, the fixes is here: https://github.com/dagger/dagger/pull/9459
I'm waiting for a review and it will be available for the next release 🙂

[fathom obsidian]

Anyone from the TS SDK devs think about using SWC to do the compilation work in the SDK? When I use dagger call it takes over 30 seconds for the TS module to initialize with each change of my module's code, which makes the DX a bit of a patience tester. SWC could speed the recompilation up, I believe.

[indigo sinew]

That's a very interesting option!

However I'm not sure it fixes the problem, most of the time spent during the initialization is dedicated to downloading dependnecies.

/cc @raven pier since you're working on speeding up the TS SDK runtime, maybe you can explore SWC, after improving deps caching, that can maybe be a quick win 🙂

Maybe we could actually bundle the TS SDK as a SWC bundle, and so we would just have to recompile the user code, that would help us skip most of the dependency downloading

[indigo sinew]

Do you know if it's possible to bundle a TS project with another one as dep?
Right now any Typescript module will depend locally on ./sdk, if we can bundle it and them compile the 2 bundle together we would achieve nice performance improvement

[fathom obsidian]

Are dependencies downloaded again for each change of code in my module?

[indigo sinew]

It should be cached but sometime it's not, we're trying to figure out why since

[fathom obsidian]

I can take some time to look into this myself. But, I'm not sure I get the mechanics of the CLI with the SDK. I'll see what I can come up with.

[indigo sinew]

All the SDK runtime logic is done there: https://github.com/dagger/dagger/blob/c811cc8b23c6398b4bf3b3ea358733759b9b9257/sdk/typescript/runtime/main.go#L105

[fathom obsidian]

I'm fairly new to Go, so please bear with me. Is my understanding correct that with every code change, the whole process of generating the client, pulling a container, installing dependencies and running the code is carried out and there is a reliance on the container build steps to be cached to "speed up" this process?

There is an if statement in CodegenBase, with the boolean installDep to install dependencies. This is hard coded to true in the call to the t.CodegenBase function in the ModuleRuntime function.

I'm guessing and as mentioned above, the container building cache step for the dependency installation is being counted on here, to skip the step. In other words, do the installation no matter what, and if the cache is already there for the step, it is used. It seems that isn't working properly. Or it is working, but code changes in the module invalidate the installation cache too, for some reason? 🤔

Couldn't there be some algo to basically do a checksum of the package.json on the first call to Dagger, and compare each further checksum for a difference and use that to trigger a reinstall of deps i.e. not depend on the container cache, but rather the code itself? I'm just spitballing here. 😄

[raven pier]

we push the checksum-the-package-json step down to buildkit as far as i understand it, so if you're seeing deps re-pulled for any code change, it's more like "installDep" thinks it depends on the whole module (or sourcedir, not sure the exact context here) rather than just the package.json

i have been putzing around in typescript/runtime/main.go myself, trying to either compress and bundle the base dependencies into the engine image or parallelize some of the steps, but i can also take a look and see if we're innappropriately invalidating cache. @pearl forge 's also got some work in the pipes to improve module cache invalidation, so possibly a moving target here

[fathom obsidian]

Ok. Thanks for the reply. Well, I'll leave it up to you experts then. Just know, to me, a much better dev feedback loop would need a 3-4 second module initialization timing (at worst) and not the 30-40 seconds I'm currently seeing with a dagger run. 😊 I hope you can find a 10x improvement somewhere.

so if you're seeing deps re-pulled for any code change

I'm also not certain the reinstallation of dependencies is actually the problem, but can only think from my TS experience, that a reinstallation is the only thing that might be causing the slowness. 🤷🏻

[kindred umbra]

Is there a way to customize which tsconfig that dagger uses?
I have a couple of ts files that I will be injecting into a container to run it there, however tsx is failing because it can't find some of the dependencies.

call constructor: process "tsx --no-deprecation --tsconfig /src/dagger/tsconfig.json /src/dagger/src/__dagger.entrypoint.ts" did not complete successfully: exit code: 1

I could just exclude them from the tsconfig.json that lives in the root, but that makes editing that file tedius since it doesn't konw about any imports

[indigo sinew]

Sadly no it’s not supported yet, but feel free to open an issue and I’ll try to add supports for that 🙂

[raven pier]

@fathom obsidian and i DM'd a bit and even version-matched v0.15.2 i cannot repro the false-positive-module-init-cache-invalidation... bout to try with a local colima k3s and the helm chart

and i can't repro in k3s either 😦 @fathom obsidian are you doing anything different from the basic k8s hello-world setup? like if you get 2 different engine pods it might look like cache invalidation, or maybe if you've modified volume configuration? either of those could look like cache invalidation for module code changes

[fathom obsidian]

@raven pier

maybe if you've modified volume configuration

What is this or rather how would I inadvertently do it?

are you doing anything different from the basic k8s hello-world setup?

I only have one engine pod. The engine is communicating over the buildkitd socket to my development pod via _EXPERIMENTAL_DAGGER_RUNNER_HOST. It is not using kube-pod, as written in the docs. I can try that as a change.

The two pods are in different namespaces, but on the same node and because of the socket, I was imagining namespace permissions are bridged because of it. If I am wrong, let me know.

[fathom obsidian]

So, I tried the kube-pod connectivity and it didn't work at all. No connection to the engine. I reverted back to the socket.

[fathom obsidian]

@edgy hollow - Let's continue here, as it is where my main issue is being discussed.

This is a trace of a run which ran quickly.

https://dagger.cloud/m8a/traces/adcd500a48a1adde1ad507427354eeda

I hope this is the right way to share the trace. If not, let me know.

This trace is where I made a change (added text to a code comment) and the run took much longer.

https://dagger.cloud/m8a/traces/8cbe0068681062878cdd51ac65c91853

I personally can't make rhyme or reason why it is taking longer now seeing the trace details.

One other note, I did get faster timing twice with two changes of the module code and I was like, OMG it's working now. WTF? But, after the third attempt, the longer timing for each change came back. 🤷🏻

[fathom obsidian]

Friendly bump. 👆🏻 😊

[fresh moss]

Hi all, new to the server and to using dagger.

Am just trying out a few things and being a bit silly but I cannot seem to get my function to always show messages I am writing to the console.log. Is there a separate function I need to call to always make it log out to my terminal? I have found if I add the -v option to the dagger call command it does log some of the data I put to the console but not all of it.

[indigo sinew]

Hey! Welcome in the Dagger community. I'm Tom, maintainer of the TypeScript SDK 🚀

The internal logging of the function is not displayed without -v or -vv, another way is to use https://dagger.cloud to check the run of your functions.

[fresh moss]

Thanks for the above Tom. I’ll check out the cloud platform 👍

[timber eagle]

https://youtu.be/pNlq-EVld70?si=45aPnmtD2MeFt_Jh

[fresh moss]

Just trying out the monorepo module setup. I have a high level module that is calling a function from a submodule. I need to mount a directory and am currently using @argument({ ignore: ["**/node_modules/**", "**/dist/**", "**/dagger/**"] }) to make sure these directories and files are not copied into the container. I would like this to be declared in the submodule but it appears that this decorator is only applied/referenced in the main function. Is that correct?

[indigo sinew]

Yeah the filter applies only when calling the function 😄

[sterile shuttle]

https://nodejs.org/en/learn/typescript/run-natively#running-typescript-natively as mentioned here
Since v23.6.0, Node.js enables "type stripping" by default. And nodejs latest version is 23.11.0, so are we following this and making our typescript SDK more powerful in TS only manner and ditching JS from code?

[indigo sinew]

It's actually the opposite that will happen, I'm working on a way to bundle the whole SDK in a single file to simplify dependency management & make the sdk directory comitable. But the bundle needs to be in js for now.

Also, we only rely on LTS version, so we'll wait for the LTS one 🙂

@bright mountain I think this may make you happy:

➜  node git:(main) ✗ du -skh sdk
4.5M    sdk
➜  node git:(main) ✗ ls sdk
client.gen.ts core.d.ts     core.js       index.ts      telemetry.ts

(it works for the quickstart [tested on node/bun for now)), I'm running the integration tests to see if there's any issues, but that's promissing 😄
I also need to find a nice way to make it compatible with all current module so we move to the bundled version smoothly when we release that

[fathom obsidian]

Nice to see continuing work in the TS sdk. Was anything found or done on the initialization speed?

[indigo sinew]

It will indeed impact the initialisation speed 🙂
since the 15second install step will simply disappear

Should be more around 8-10 now when I tested 🙂

[fathom obsidian]

Cool. Looking forward to that for sure! 🙂

And maybe with the new TS compiler in Go, it might shave a second or two off too.

[indigo sinew]

Yeaaah! I'll post some updates on this channel when things are merged

[indigo sinew]

https://github.com/dagger/dagger/pull/10094 Pr opened 😄

@solar shuttle @glad kraken If you guys have some time to review it today, that would be amazing.
I'm working on some benchmarks to see the different between bundle vs no bundle

[sterile shuttle]

https://github.com/pnpm/pnpm/pull/9358

[indigo sinew]

Benchmark for the bundling: https://github.com/dagger/dagger/pull/10094#issuecomment-2786296756

[fathom obsidian]

Those are really nice improvements. Well done!

[indigo sinew]

I'll try to get that merged next release, I still need to do couple of tests around compatibility with current modules to ensure smooth transition

[mint ibex]

Hey I just wanted to confirm my assessment on this dagger code:

  @func() 
  GoContainer(source: Directory): Container {
    return dag.container().from("docker.io/library/golang:1.24")
      .withDirectory("/go/src", source, { include: ["go.mod", "go.sum"] })
      .withWorkdir("/go/src")
      .withExec(["go", "mod", "download"])
  }

  @func()
  GoSrcCode(source: Directory): Container {
    return this.setupGoContainer(source)
      .withDirectory("/go/src", source, { exclude: ["go.mod", "go.sum"] })
  }

  @func()
  Build(source: Directory): File {
    return this.copyGoFiles(source)
      .withMountedCache("/root/.cache/go-build", dag.cacheVolume("go-build-cache"))
      .withExec(["go", "build", "-o", "unkey-api", "./build/api/main.go"])
      .file("/go/src/unkey-api")
  }

My Goal is to minimise the build time.
So should I move it code of GoSrcCode Build or GoContainer ? I think I should do move it in Build as it would allow at least 1 Node(function) will be cached and there will be 0, if I moved it GoContainer

Is my argument correct, by Dagger's Design?

Also, does dagger treat each instruction like withExec, withMountedCache like layers just like docker does?

[sterile shuttle]

https://github.com/yarnpkg/berry/pull/6752

[indigo sinew]

Hello 😄

Is my argument correct, by Dagger's Design?

It's correct but it could be better, I see you're using include/exclude when you copy directory so you're doing a "post call filtering" but what you could do is a "pre call filtering" so only the files you are using will be copy in the engine.
See that doc for an example: https://docs.dagger.io/api/filters/#pre-call-filtering
You can use it for both including ( ignore: ["!go.mod", "!go.mod"]) or exclude files (ignore: ["**.md"])

Also, does dagger treat each instruction like withExec, withMountedCache like layers just like docker does?

Yes exactly! That's why the order of operation is very important 🙂

In your case, I would try to use a constructor to get my source code: https://docs.dagger.io/api/constructor
And then reuse that across my different functions.

Maybe then depending on your usage, you could convert the function GoContainer to set a field in your module's object so you can reuse it in your build function so for example:

@object()
export class Example {
  @field()
  source: Directory

  @field()
  ctr: Container

  constructor(
   // I use ignore here to only include files that are useful for my Go project
   // And I use defaultPath so I don't have to specify an argument in the CLI, it will directly take the git directory my module is in.
   @argument({ defaultPath: "/", ignore: ["!**/*.go", "!go.sum", "!go.mod"]})
   source: Directory
  ) {
    this.codebase = source
    this.ctr = dag.container()
  }

  @func()
  container(version: string = "1.24"): Example {
   this.ctr = this.ctr.from(`docker.io/library/golang:${version}`)
     .withDirectory("/go/src", this.source)
     .withWorkdir("/go/src")
     .withExec(["go", "mod", "download"])   
 
   return this
  }

  @func()
  build(): File {
    return this.ctr.withMountedCache("/root/.cache/go-build", dag.cacheVolume("go-build-cache"))
      .withExec(["go", "build", "-o", "unkey-api", "./build/api/main.go"])
      .file("/go/src/unkey-api")
  }
}

So on usage I would just do dagger call container build or dagger -c 'container | build'

The steps are layered correctly so you can benefit from cache there (Actually I would also add a cache layer on the go mod download for better dep caching) 😄

Let me know if you have any questions

We also have a couple of Golang modules already made by our community, maybe you can get some inspirations there: https://daggerverse.dev/search?q=golang 🙂

[mint ibex]

Wow thanks. I am trying to wrap my head about the constructor part. This is something new.
Thanks again for all the help!

[mint ibex]

Hey @indigo sinew :
I implemented my functions into Class, constructors:

@object()
export class GolangProject implements Project {
    source: Directory
    ctr: Container
    buildCache: CacheVolume
    packageCache: CacheVolume;
    readonly projectDir = "/go/src/";
    constructor(
        @argument({ defaultPath: "/", ignore: ["!**/*.go", "!go.sum", "!go.mod", "dagger"] })
        source: Directory
    ) {
        this.source = source; // Copy the entire monorepo to get the .git directory
        this.ctr = dag.goreleaser({ version: "latest" }).ctr();
        this.buildCache = dag.cacheVolume("go-build-cache") // ~/.cache/go-build (Linux)
        this.packageCache = dag.cacheVolume("go-pkg-cache")
    }


    @func()
    setup(buildDir: string,): GolangProject {
        const workdir = buildDir ? this.projectDir + buildDir : this.projectDir;
        // Assuming all the package manager files are in the root directory
        this.ctr.withDirectory(this.projectDir, this.source)
        .withMountedCache("/root/.cache/go-build", this.buildCache)
        .withMountedCache("/go/pkg/mod", this.packageCache)
        .withWorkdir(workdir)
        .withExec(["go", "mod", "download"])
        return this
    }
}

@object()
export class Dagger {

    @func()
    buildGoProject(source: Directory, buildDir: string): Container {
        const project = new GolangProject(source);
        return project
            .setup(buildDir).ctr.terminal()
    }
}

Now .withWorkdir(workdir) should have change the directory and stayed that way.
But When I go into the terminal in func buildGoProject() then I am in the /go dir, I am confused why is this happening?

I applied pre-filtering to the same code. But everytime I change my dagger code. The filesync happens on every call. Shouldn't this be fast, because it had already copied the code?

[mint ibex]

nvm, this was a very stupid mistake of mine.
Had to replace this.ctr.withDirectory with this.ctr = this.ctr.withDirectory. Now its working for this code.

[indigo sinew]

Very nice yeah! Yeah the state needs to be stored in your variable to be populated to next functions haha.
The other way is to chain functions together 🙂

[carmine jolt]

Small PR to improve typescript configs: https://github.com/dagger/dagger/pull/10203

[bright mountain]

Thank you @carmine jolt ! You're on a roll this week 🙂 We appreciate all the contriutions

[indigo sinew]

I still see the tests failing, let me know when you have finish updating the PR so I can give it another review 😉

[carmine jolt]

Will do! I've been out of town so haven't had a chance yet, but will get to it as soon as i can and will let you kno!

[indigo sinew]

No pressure haha, you're doing great 🚀

[carmine jolt]

@indigo sinew tests are green now!

[indigo sinew]

Will have a look soon, I'm merging https://github.com/dagger/dagger/pull/10094 first 😄

[indigo sinew]

Merged 🚀

[rotund ember]

@quaint plover here's the main place to chat about typescript

[sterile shuttle]

https://deno.com/blog/add-jsr-with-pnpm-yarn

[carmine jolt]

Why was the path to client.gen.ts changed here? https://github.com/dagger/dagger/commit/312255260f8d640c8260b72f51d4487989228987#diff-3f816215b584a031057b440089ceb90784c3d4257816d7facfa3dcec14fe0a94R5731
After updating to 0.18.5, all of the external modules are no longer on Client:

This seems to be because import { dag } from @dagger.io/dagger still points to sdk/src/api/client.gen.ts

So now there are 2 client.gen.ts files

ok so I think importing modules is completely broken in 0.18.5

[edgy hollow]

cc @indigo sinew 😬

[edgy hollow]

@carmine jolt how about if you remove the generated code and run dagger develop again?

[carmine jolt]

That's what I did

[edgy hollow]

did it fix the issue for you?

[carmine jolt]

no

I think I'm going to have to revert back to 0.18.4 to make it work

[edgy hollow]

did you change the paths in the tsconfig.json?

[carmine jolt]

no - all I did was upgrade to 0.18.5, delete sdk/ and run dagger develop

[edgy hollow]

o130|marcos:tmp/tstest (⎈ |kind-kind)$ cat tsconfig.json
{
  "compilerOptions": {
    "target": "ES2022",
    "moduleResolution": "Node",
    "experimentalDecorators": true,
    "strict": true,
    "skipLibCheck": true,
    "paths": {
      "@dagger.io/dagger": ["./sdk/index.ts"],
      "@dagger.io/dagger/telemetry": ["./sdk/telemetry.ts"]
    }
  }
}

^ that's how they look in v0.18.5

[edgy hollow]

yeah.. something's broken there when upgrading

I'd assume it didn't work for you because tconfig.json had some 0.18.4 values

[carmine jolt]

I just updated tsconfig.json to what you posted and regenerated sdk/ dir - still doesn't work

[edgy hollow]

checking myself now

[edgy hollow]

ok.. in a new project without dependencies everything seems to work ok. Heres' what I did:

dagger184 init --sdk typescript
dagger184 call ... # works
dagger develop #dagger 18.5
dagger call ... # works in 18.5

not sure if it might be related to installed modules since you have a more complex project on your end

[carmine jolt]

ok I'll play with it hang on

[edgy hollow]

gotta go. I'll be back later 🙏

[carmine jolt]

okay I figured it out - I didn't read the migration steps and didn't remove @dagger.io/dagger=./sdk from package.json 🤦‍♂️

maybe we should add BREAKING to the so that people skimming the changelog pay more attention

(or i should pay more attention to changelogs)

[edgy hollow]

No worries! It happens to us all. Seems to be it should be a breaking change indeed if some action is required from users

[edgy hollow]

According to the notes it should upgrade automatically by calling dagger develop. Wondering why that didn't work for you

[bright mountain]

what's the nature of this breaking change?

[edgy hollow]

Shouldn't be a breaking change. Still trying to get a repro of how it supposedly breaks. But seems related to https://github.com/dagger/dagger/pull/10094

[indigo sinew]

It’s not a breaking change, it’s either you configure the SDK to be bundled or local, but you can’t do a mix haha

I’ll do some tests on Monday but normally it shouldn’t break if you keep the local library

[carmine jolt]

idk, all I did was update the dagger version and tried running dagger develop I can try to get a repro later

[indigo sinew]

Hmmm I have integration tests for all of this, that’s strange that you have hit this issue, should have been caught by tests

[carmine jolt]

I think it might have been that I deleted sdk/ directory in one module, but not a sub-module

[indigo sinew]

Let me know when you can create a repro! If there’s a bug I’ll fix it as fast as possible

And also let me know if you have any feedbacks about the bundled version, the goal is to make things better so if it doesn’t in your case, I’ll be happy to hear your opinion 😄

[carmine jolt]

I’ll let you know! One thing I will note is that I always delete the generated .gitignore because I don’t want to have run dagger develop 15 times any time I clone my repo to a different machine, and would rather just commit the sdk dir

[indigo sinew]

Interesting! That’s something I was discussing with Helder last week, technically the bundled version could be commutable

However I think we would need some kind of logic in the CLI to bypass the generation, because right now the runtime will always copy the bundled lib etc no matter the state of the SDK dir

So it doesn’t really impact the run to commit the SDK or not for now, but we could maybe change that behaviour at some point, like a —frozen or something, not sure yet

[carmine jolt]

All I really care about is that types work when I’m modifying .dagger/src/index.ts

[indigo sinew]

Normally you shouldn’t need to run dagger develop more than when you clone the repo, on other actions like adding a new deps, the client will auto regenerate

Or if you update a module, then yeah dagger develop is needed but that’s pretty much it

[carmine jolt]

Right - but still we’re talking about like 15 modules where I have to run dagger generate after cloning the repo.

I’m used to only needing to “pnpm install” in my monorepo to be able to edit everything in the repo, so not committing the generated files would mean 16x more commands to get the same

[indigo sinew]

I see, do you have a module that imports all your local’s one? I think that running dagger develop on the main module will also run it recursively to others

And if not, I think there’s an issue about that somewhere

[carmine jolt]

Oh really? I’ll have to try that. Cause yeah I do have one top-level module to call all the others

[indigo sinew]

Give it a try, I’m not sure it’s done but I’m sure I saw this mentioned somewhere in an issue

[carmine jolt]

Thanks - will try when I get back to my laptop

[carmine jolt]

tested - dagger develop definitely does not run on dependent modules

For now I may just write a script to run bun install && dagger develop in all my modules
edit: wrote the script!

[carmine jolt]

now that I can setup the repo in one command, I feel better about removing the sdk files

[glad kraken]

dagger develop --recursive should for all the local dependencies recursively

[carmine jolt]

oooo thanks - that worked! 🎉

[maiden eagle]

Is there a difference in coverage or update frequency between the TS and Python SDKs? I want to consolidate my existing scripts and need to pick a language.

[indigo sinew]

Hey, main maintainer here 👋

We try to be as reactive as possible and keep consistency between all officials SDKs (Go, Ts, Python).
I would suggest to choose your language based on your / your team personnal preferences and what you're more used to work with.
Because you're using code to develop pipeline, script, workflows etc, being fluent with the language let you creates more efficient and scalable pipelines 😛

The 1 big difference is performances, which will impact your feedbacks loop when you develop your module (specially when you start to have multiples one that call eachother).
Go will always be the fastest, Typescript and Python are slower and have similar performances (TS might be a little bit faster depending on your setup but it's minimal).

If you can tell me more about what you plan to build exactly, I can give you more guidance 😄

---

@pearl forge @edgy hollow @raven pier If one of you guys have a moment to review this PR today, that would be amazing: https://github.com/dagger/dagger/pull/10299
It's a quick fixes for an edge case that I discovered today while I was doing a demo 😄

Failed tests are not related to the PR (Java / Git auth) so I think it's safe to approve

[hexed tree]

const denoFmtSupportedFiles = [
    ...codeFiles,
  "!**/*.astro",
  "!**/*.css",
  "!**/*.html",
  "!**/*.json",
  "!**/*.jsonc",
  "!**/*.md",
  "!**/*.scss",
  "!**/*.sql",
  "!**/*.vue",
  "!**/*.yaml",
  "!**/*.yml",
];

  @func()
  format(
    @argument({
      ignore: [
        "*",
        ...packageFiles,
        ...denoFmtSupportedFiles,
      ],
    }) directory: Directory,
  ): Container {

Is this expected to work, because I'm having some problems with it 🙂

[sterile shuttle]

Just to add in context https://deno.com/blog/v2.3

[indigo temple]

Upgraded from v0.18.0 to v0.18.5 and my local module is not seeing codegen with the dag client?

[drowsy oracle]

This sounds like https://github.com/dagger/dagger/issues/10300
It could be useful to add another repro if you have one

[indigo sinew]

No, it needs to be inlined in the decorator because the TypeScript compiler API introspect the static version of the file, so it doesn't resolve runtime values :/
If you have some experience with the compiler API and know how to resolve them properly, I would be super happy to fix that

[robust compass]

I've noticed something with the TS SDK that I'm just trying to understand. The Daml class is the module root. This works as expected, but if I use this.daml.container in the inspect function, I get the following error: Error: this.daml.cont(...).withFile is not a function

@object()
export class Daml {
  @func()
  container: Container

  // ...
}

@object()
export class Archive {
    daml: Daml
    container: Container
    name: string
    version: string
    file: File

    constructor(daml: Daml, name: string, version: string, file: File) {
        this.daml = daml
        this.container = daml.container
        this.name = name
        this.version = version
        this.file = file
    }

    @func()
    filename(): string {
        return `${this.name}-${this.version}.dar`
    }

    @func()
    async inspect(output?: string): Promise<string> {
        const args = ["daml", "damlc", "inspect-dar"];
        if (output) {
            args.push("--" + output);
        }
        args.push(this.filename());

        return await this.container
            .withFile(`/home/daml/${this.filename()}`, this.file)
            .withExec(args)
            .stdout();
    }
}

[robust compass]

TypeScript SDK Container Field

[earnest tusk]

Hey folks,
Sorry it's really broad and noob question but I'd like to start to use Dagger replace with Github actions workflow for one my nodejs application..
Can someone guide me where should i start? 🙂

1- I'd like to write my own modules with Typescript but not sure should I or reinvent the wheel? Is it possible to import any module from daggerverse, if yes any example?
2- I did not fully undertand how cache replace with Github actions considering actions/setup-node interact with github actions cache? Any guidance will be appreciated.

[drowsy oracle]

Hey folks,

[kind cape]

Does Dagger support cli arguments for it's functions?

There is @arguments but that is only able to define the sources to include and what to ignore.

I thought on something like this:

  /**
   * Runs the full Continuous Deployment (CD) pipeline.
   * @param version The semantic version to release.
   * @param npmToken A secret containing the NPM token.
   * @param source The source code directory.
   */
  @func()
  async fullPipeline(
    version: string,
    npmToken: Secret,
    @argument(argumentsOptions.source) source: Directory,
    @argument({ default: false })
    skipGithubIntegration: boolean = false,
    @argument({ optional: true })
    githubRepoSlug?: string,
    @argument({ optional: true })
    githubToken?: Secret,
  ): Promise<string> {

[indigo sinew]

Hey 😄

By default any arguments of your function will be available as argument from the cli (or shell)

So if you have:

@func()
foo(version: string): ... {}

You can do dagger call foo --version "xx" 🙂

You can find more information at https://docs.dagger.io/api/arguments that explains how to make arguments optional, with default value etc

[mint ibex]

Context: #general message

To get the speed bump, I just need to upgrade dagger right?
Current: dagger v0.18.12 (docker-image://registry.dagger.io/engine:v0.18.12) darwin/amd64

[indigo sinew]

It should effectively increase your module’s performance 🙂
if not, please open an issue with some dagger traces so I can have a look

[random coyote]

Hey @brazen wave , I'm confused about @enumType and the official TS docs: https://docs.dagger.io/extending/enumerations. How are we supposed to expose an enumeration ?

/**
 * The definition of the `@enumType` decorator that should be on top of any
 * class module that must be exposed to the Dagger API as enumeration.
 */
declare const enumType: () => (<T extends Class>(constructor: T) => T);
/**

[indigo sinew]

The first enum implementation relied on decorated classes because the introspection system didn't allow using the TypeScript enum keyword since that cannot be decorated.
Then I added to the introspector the capability to follow references of the main module declaration instead of naively parsing decorators so we could switched to native TypeScript enumeration (and also support interface).
This enumType decorator should be deprecated since we don't use it anymore

[random coyote]

since that cannot be decorated.
So ... following these changes, is there a way to decorate enums in ts ? As part of @deprecated, I wanna be able to deprecate enum members. I circumvent the issue by accepting JSdoc comments ?

That's my current pseudo-code in TS:

import { dag, object, func, argument, GitRepository } from "@dagger.io/dagger"

@object({
  // planned deprecation hook once we wire it through
  deprecated: "This module is deprecated and will be removed in future versions.",
})
export class Test {
  private _legacyField = ""

  @func({
    deprecated: "This field is deprecated and will be removed in future versions.",
  })
  legacyField(): string {
    return this._legacyField
  }

  @func()
  async echoString(
    @argument({
      deprecated: "Use 'other' instead of 'input'.",
    })
    input: string,
    other: string,
  ): Promise<string> {
    return input
  }

  @func({
    deprecated: "Prefer EchoString instead.",
  })
  async legacySummarize(note: string): Promise<LegacyRecord> {
    return { note }
  }

  @func()
  useMode(mode: Mode): Mode {
    return mode
  }
}

/** @deprecated This type is deprecated and kept only for retro-compatibility. */
export type LegacyRecord = {
  /** @deprecated This field is deprecated and will be removed in future versions. */
  note: string
}

export enum Mode {
  Alpha = "alpha", // @deprecated alpha is deprecated; use zeta instead
  /** @deprecated beta is deprecated; use zeta instead */
  Beta = "beta",
  Zeta = "zeta",
}

Basically, alias types + enums don't support decorators. Any idea on how to circumvent that ?

[indigo sinew]

Hmmm that’s really complex…
Maybe with a jsdoc comment but I didn’t created any logic for that

I think you can leave it for now, don’t block the implemention bc of that, you can still add support for @deprecated on top of @enumtype

Or ideally, don’t expose a @deprecated decorator but instead only rely on JsDoc @deprecator since that’s the native way

[random coyote]

Opening a small PR for that

[random coyote]

Mmmh, I’m a bit confused :

• When you say “add support for @deprecated on top of @enumtype”, do you mean literally keep using the old @enumType decorator and extend it with deprecated, or something else?
• And when you say “only rely on JsDoc @deprecated since that’s the native way”, is that basically what I showed here?

/** @deprecated beta is deprecated; use zeta instead */
Beta = "beta"
Alpha = "alpha", // @deprecated alpha is deprecated; use zeta instead

My first idea was to just extend @func, @object, @argument with a deprecated argument (the same way you do it for defaultPath), but enums and type aliases can’t take decorators anyway. So are you suggesting the UX should be:

• decorators for classes/functions/args
• JSDoc for enums/types

Or, are you saying that everything should be a JSDoc @deprecated ?

[random coyote]

Maybe with a jsdoc comment but I didn’t created any logic for that
I'll try to do it ahah, it's a great excuse to ramp up on the ts SDK 😝

[indigo sinew]

Yeah, I’m saying deprecated should be with jsdoc since that’s the usual way to deprecate function in typescript

[random coyote]

perfect !

[random coyote]

https://github.com/dagger/dagger/pull/11076

[oblique fjord]

ELI-IAMan00b

When do I need to run dagger develop --sdk=typescript?

If I install @dagger.io/dagger from npm, do I still need to generate the SDK? Seems to work without it.

Is there a bare minimum, Hello World example repo with a TypeScript setup?

[bright mountain]

dagger develop is needed when you develop Dagger Functions - those are built and run by the Dagger Engine (as API extensions), not on your host system. So there can be drift between your local dev environment, and the actual build and runtime environment of your functions. dagger develop keeps your dev environment in sync with the actual build & run environment, so that eg. your IDE auto-completion works etc.

If you're not developing Dagger Functions, but rather a regular native program (like a custom CLI for example), and you are only importing the Dagger client library (@dagger.io/dagger) then you don't need dagger develop

[oblique fjord]

Hey Solomon, thank you for taking the time.

But I’m still unclear.

What’s the difference between a custom CLI and Dagger Functions?

I’m writing some functions in TypeScript with a decorator @func. is this a dagger function?

I’m not intending to publish it. It’s a CI plan.

[bright mountain]

Yes that is a dagger function.

In that case, you should not install @dagger.io/dagger. Instead you should:

• Run dagger init --sdk=typescript in the directory that you want to use as the root of your dagger module. Typically this would be the directory that corresponds to the software component you're writing CI functions for. Could be the root of your repo, or a subdirectory if you're in a monorepo. dagger init --sdk=typescript will generate boilerplate with sample functions, and all the bindings you need. Then you can use the special dag variable directly in your code

• dagger develop is only needed if you need to re-generate the bindings in your local dev environment. In practice you don't need it very often.

If you haven't already, I recommend following the Typescript quickstart for a CI module: https://docs.dagger.io/getting-started/quickstarts/ci?sdk=typescript

[oblique fjord]

Would I commit the generated code?

[bright mountain]

You can - but by default dagger init generates a .gitignore that prevents it. It's safe to change that .gitignore, it's just a sane default.

I know management of generated files is a matter of preference.

However, note that regardless of how you manage the generated files they are always re-generated at runtime. In other words, the engine discards your copy of generated files. They are only for the benefit of your local dev environment (IDEs etc)

[oblique fjord]

I’m thinking about developers who need to use this. If they git checkout and there’s no pre generated code they need to do an extra step each time?

[bright mountain]

If you don't commit them (ie. the default), after a fresh commit you will need to run dagger develop to re-generate them for your IDE

[bright mountain]

Right, yes that's dagger develop (to get IDE auto-completion etc). You don't need to run dagger develop for the functions to run. dagger call will always work from a fresh checkout.

If you prefer that a fresh git checkout have everything ready for the IDE, then you should change the .gitignore so that everything gets committed

either way works just fine

[oblique fjord]

Ah gotcha. So only if they want to work on the dagger setup they need to codegen. If only to use they do not.

Thank you 🙏

So the minimal viable setup then is:

package.json with type module and typescript package

A tsconfig with path mapping to codegen

A dagger module? Not sure what’s the nomenclature of the entry point dagger TypeScript code.

I think that’s all, right?

[bright mountain]

For a dagger module, the minimal viable setup is the output of dagger init and nothing else. That will include:

• dagger.json which marks the root of the dagger module
• A source directory for your typescript implementation. You can configure it with dagger init --source, but by default if the directory has pre-existing files, it will place the source in .dagger. Then in there, it's a typical typescript project layout (which you can customize if you want)

[oblique fjord]

I think {npm|pnpm|yarn} installs should add a flag to only install prod deps.

We are adding ESLint and other tooling to dev deps, but I think Dagger engine does not need it to execute the module code. And it slows down the execution by many seconds having to instal extra dev deps.

Happening here: https://github.com/dagger/dagger/blob/5b07b0b4b4208b5009866e7219a72d0957213c44/sdk/typescript/runtime/module.go#L214

[glad kraken]

@indigo sinew does this seem reasonable?

imo, we should be defaulting to only installing prod deps

[indigo sinew]

Yeah that’s a side effect of the old dagger typescript implementation that was installing the SDK as a dev dependencies.
I never took the time to change it after we change the SDK dep location, I’ll do a PR to integrate that

[indigo sinew]

Another reason that I remember now is that the default package.json has no dev dependencies: https://github.com/dagger/dagger/blob/main/sdk/typescript/runtime/template/package.json

[gusty sail]

hi all! this has probably been asked before, but how do I return a folder containing build artifacts and/or test results (might be 2 different folders) to the build agent from dagger?

[copper trout]

It seems like the typescript function caching configuration isnt work properly? I'm getting a type error on @func({ cache: "never" }), func is expecting just a string for alias

[bright mountain]

@indigo sinew 🙋 👆

[indigo sinew]

Hey I took some time to dig into that, you're right about that type error, however you can still call your functions and it should execute fine.

I opened a PR to fix that type error so it will be resolved on next release 🙂

[bright mountain]

Someone on Hacker News compared our Typescript DX to AWS-CDK:

https://docs.dagger.io/cookbook/services?sdk=typescript
Still looks like "a circa-2000s Java builder API" and doesn't look like pleasant / declarative / idiomatic TypeScript, which is what aws-cdk pulled off.
Genuinely impressively (imo), aws-cdk intermixes "it's declarative" (you're setting up your desired state) but also "it's code" (you can use all the usual abstractions) in a way that is pretty great & unique.

I know it's not apples to apples, and this person hasn't actually used Dagger. But, I would like to know what you all think of that comparison. Is aws-cdk more ts-native, in a way that we should copy?

[indigo sinew]

I'm not sure the DX will work with our DAG system and I don't think aws-cdk feels more native than our current way. It's simply 2 different ways.

This is an example of a aws-cdk script, basically instead of calling dag.xxx, you create an actual TypeScript object with new. Then the resolution is done when calling syncth, while this happen when you return on dagger.

#!/usr/bin/env node
import autoscaling = require('aws-cdk-lib/aws-autoscaling');
import ec2 = require('aws-cdk-lib/aws-ec2');
import elb = require('aws-cdk-lib/aws-elasticloadbalancing');
import cdk = require('aws-cdk-lib');

class LoadBalancerStack extends cdk.Stack {
  constructor(app: cdk.App, id: string) {
    super(app, id);

    const vpc = new ec2.Vpc(this, 'VPC');

    const asg = new autoscaling.AutoScalingGroup(this, 'ASG', {
      vpc,
      instanceType: ec2.InstanceType.of(ec2.InstanceClass.BURSTABLE4_GRAVITON, ec2.InstanceSize.MICRO),
      machineImage: ec2.MachineImage.latestAmazonLinux2023({
        cpuType: ec2.AmazonLinuxCpuType.ARM_64
      })
    });

    const lb = new elb.LoadBalancer(this, 'LB', {
      vpc,
      internetFacing: true,
      healthCheck: {
        port: 80
      },
    });

    lb.addTarget(asg);
    const listener = lb.addListener({ externalPort: 80 });

    listener.connections.allowDefaultPortFromAnyIpv4('Open to the world');
  }
}

const app = new cdk.App();
new LoadBalancerStack(app, 'LoadBalancerStack');
app.synth();

I think Dagger and aws-cdk notations can't be the same, with Dagger you create GQL query that will be resolve to return a state/result, with aws-cdk you declare resources and that's it so function chaining isn't really useful.

I'm not sure how we could take inspiration on AWS CDK, potentiellement we could find a way to let people create resources using new instead of dag but that's pretty much it.

If I take the user example and convert it to AWS CDK we would have something like:

class MyModule {
  @func()
  httpService(): Service {
    const container = new Container({
      from: "python",
      workdir: "/srv",
      files: [new File("index.html", "Hello world!")],
      exposedPort: [8000],
    })

    return new Service(
      container: container,
      args: ["python", "-m", "http.server", "8080"]
    )
  }

[oblique fjord]

I read that comment and I resonate with it very much.

For me it’s about using actual expressive code to declare infra concerns.

As an software engineer now a platform engineer I’m a lot more comfortable writing TypeScript than maintaining YAML.

[oblique fjord]

What is live to see is the ability to package solutions into npm packages and release to a registry. Such that consumers can just npm install a dag

CDK is based on a phenomenal piece of tech JSII. I think Dagger could use that too. But of course it’d be a big rewrite so unlikely

[bright mountain]

Yeah jsii is nice but inferior to http/graphql for our particular use case (which is different from cdk)

dagger is available as a npm dependency already so you are free to import it and build your own js libraries on top, then distribute them on npm. Of course you won't get the cross-language features that way

[bright mountain]

@drowsy oracle I'm messing witih typescript/docusaurus builds in modules (dogfooding modulesV2), and struggling with super slow builds that don't benefit from cache volumes... I remember you saying "don't put node_modules in a cache volume" does that still stand? I have node_modules/.cache in a cace volume but that doesn't seem to help much

[drowsy oracle]

The risk of putting ./node_modules in a cache volume is that the dependencies are used at runtime and not just at build time. So if another session modifies that cache volume its catastrophic. The proper place for the cache volume would be npm's cache at /root/.npm, or specify it like we do here: https://github.com/kpenfound/dag/blob/main/node/main.dang#L63-L67
The problem is that then npm install will actually copy from that cache volume. In the past, with the ridiculous amount of files npm packages tend to have, the copy is only about 50% faster than just downloading them in the first place

[bright mountain]

I already have these mounted globally:

      .withMountedCache(
        "/root/.npm/_cacache",
        cacheVolume("npm-cache"),
      )
      .withMountedCache(
        "/usr/local/share/.cache/yarn",
        cacheVolume("yarn-cache"),
      )
      .withMountedCache(
        "/root/.local/share/pnpm/store",
        cacheVolume("pnpm-cache"),
      )

👆 mmm should I protect those 3 too with LOCK ?

Plus <app>/node_modules/.cache namespaced by app

[drowsy oracle]

mmm should I protect those 3 too with LOCK

I think concurrent access to the cache directories is safe

[bright mountain]

OK, so I already have roughly the optimal cache volume setup?

[drowsy oracle]

Yup, thats how I'd recommend doing it. With a lot of dependencies, the npm install step can still take several seconds even if its using a populated cache volume which is unfortunate, but its the safest option

/root/.npm/_cacache looks like it should just be /root/.npm though

I'm also not 100% sure with some package managers if you need to set a configuration to make sure they actually write to the cache. I've seen behavior before with yarn where I suspected it was not copying to cache, but haven't dug further

[bright mountain]

Question for typescript-savvy daggernauts... is console.log() from the module code supposed to show up in the output of dagger call?

Re: #1491911173115875489

[bright mountain]

@indigo sinew I was wondering if there are any dependencies in the typescript SDK that we should freeze or remove because of the ongoing chaos in the npm ecosystem?