Typescript: Request Error: getaddrinfo ENOTFOUND repo.yarnpkg.com

[quaint gyro]

Hi, i am currently trying Dagger with typescript in my company, I have an example repo to test Dagger with an initial setup of dagger with typescript SDK.

Currently yarn on local work, I don't have Docker on local, and with Dagger on a remote Dagger Engine I always have this error when I just run dagger call containerEcho:

PS C:\Users\MyUser\my-example-repo> dagger call containerEcho
[...]

Stdout:
invoke: failed to create codegen base: could not list rootfs entries: input: container.from.withoutEntrypoint.withExec.withEnvVariable.withMountedCache.withExec.withWorkdir.withDirectory.withExec.withExec.withExec.directory resolve: process "yarn set version stable" did not complete successfully: exit code: 1

Stdout:
Request Error: getaddrinfo ENOTFOUND repo.yarnpkg.com
[...]
info Visit https://yarnpkg.com/en/docs/cli/policies for documentation about this command.
Stderr:
! Corepack is about to download https://registry.yarnpkg.com/yarn/-/yarn-1.22.22.tgz
! The local project doesn't define a 'packageManager' field. Corepack will now add one referencing yarn@1.22.22+sha512.a6b2f7906b721bba3d67d4aff083df04dad64c399707841b7acf00f6b133b7ac24255f2652fa22ae3534329dc6180534e98d17432037ff6fd140556e2bb3137e.
! For more details about this field, consult the documentation at https://nodejs.org/api/packages.html#packagemanager

! Corepack is about to download https://repo.yarnpkg.com/4.3.1/packages/yarnpkg-cli/bin/yarn.js
error Command failed.
Exit code: 1
Command: /usr/local/bin/node
Arguments: /usr/local/lib/node_modules/corepack/dist/corepack.js yarn@stable set version stable
Directory: /src
Output:

I tried to enforce the package manager with this on ./dagger/package.json:

  "packageManager": "^pnpm@9.6.0"

I doesnt change anything.

And a working .npmrc and .yarnrc.yml on local doesnt fix anything if I copy them at the root project dir or in the dagger directory.

[celest kernel]

hey @quaint gyro! are you running the latest v0.12.2 version?

cc @regal solar

[quaint gyro]

Hi, this is happening with the 12.1. version. Do you want me to try with the latest?

[celest kernel]

@quaint gyro is this the default containerEcho function you're trying to run?

or you've made any modifications to the project?

do you have a few minutes to check it together in #911305510882513037?

[quaint gyro]

Yes that's the default funcs created with dagger init.
No modifications made.
Sorry i am not available now. 😕

[celest kernel]

does the machine where you have Dagger installed has access to repo.yarnpkg,com?

the first error I see in the logs is about not being able to resolve that address

[regal solar]

Hey, this will be fixed on we use: https://github.com/dagger/dagger/pull/7864

[celest kernel]

@regal solar what I understood is that @quaint gyro added the packageManager aftewards in order to test if they could fix the issue. I don't think that's an underlying requirement in this case 🤔

[quaint gyro]

no, i was effectively trying to fix my current issue by using packageManager.

[quaint gyro]

i can't say because that's my colleagues which have admin access to the server but * can say that we need to use some configuration with proxy and custom registry to make npm and yarn work in our company network.

[celest kernel]

ok, that's probably the issue then

can you tell your colleagues to check our docs on how to set that up? https://docs.dagger.io/manuals/administrator/proxy https://docs.dagger.io/manuals/administrator/custom-ca
happy to help also if they can't figure it out as it's not super straightforward currently

[quaint gyro]

hi @celest kernel, I just talk about that with him, proxy configuration is correctly set and for the custom certs we are using this:

          command:
            - /bin/sh
            - -c
          args:
            - "update-ca-certificates && /usr/local/bin/dagger-entrypoint.sh --addr tcp://0.0.0.0:1234"
          volumeMounts:
            - mountPath: /usr/local/share/ca-certificates
              name: foobarcompany-ca-cert
            - mountPath: /var/lib/dagger
              name: dagger-storage

[quaint gyro]

one thing i don't understand is: How could we configure yarn/npm with custom registries on remote dagger engine?

[celest kernel]

this seems correct.

[celest kernel]

do you know if the YARN/npm registry can be configured using an env variable?

how do you currently configure that?

IIRC there's a YARN_REGISTRY env variable, correct?

[quaint gyro]

currently i am using a .yarnrc.yml like this:

enableStrictSsl: false
httpsCaFilePath: "C:/Users/me/root-ca-cert"
npmRegistryServer: "http://nexus.foobar-company.io/content/groups/public-npm-releases/"
httpProxy: "http://me:[e-mail]:3128/"
httpsProxy: "http://me:[e-mail]:3128/"

and a .npmrc like this:

cafile=C:\Users\me\root-ca-cert
registry=http://nexus.foobar-company.io/content/groups/public-npm-releases/
proxy=http://me:[e-mail]:3128/
https-proxy=http://me:[e-mail]:3128/
no_proxy=*.foobar-company.io

It works for local usage of npm/yarn on my computer.

[quaint gyro]

didnt know about YARN_REGISTRY, i will check that

[celest kernel]

oh, I see. Unfortunately, I don't think we support loading .yarnrc or .npmrc at the sdk initialization phase. cc @regal solar

could you open an issue for that @quaint gyro please?

the engine has the ablity to set env variables that the SDK runtimes can use for the moment.

[quaint gyro]

yes I will open a new issue

[celest kernel]

@quaint gyro does initializaing a new typescript module work for you?

i.e: dagger init --sdk typescript

does that work in a new folder with your dagger server?

[quaint gyro]

I tried and no it doesn't works, for the same reason: Request Error: getaddrinfo ENOTFOUND repo.yarnpkg.com

[celest kernel]

@quaint gyro in the engine configuration you shared above, I don't see any proxy engine configurations: https://docs.dagger.io/manuals/administrator/proxy/.

Can you check with your team if they're setting those there?

[quaint gyro]

Sorry I forgot to mention that, here is the env vars used by the remote Dagger engine:

          env:
            - name: HTTP_PROXY
              value: http://proxy.foobar-company.io:8080
            - name: HTTPS_PROXY
              value: http://proxy.foobar-company.io:8080
            - name: NO_PROXY
              value: .foobar-company.io

[quaint gyro]

I don't know if it's always the best way to pass the custom conf for each project, in our case I think having a global custom package manager conf set at the remote Engine level is sufficient and avoid the needs for Dev teams to handle that.

[celest kernel]

this is strange... IIUC seems like the engine is not being able to talk to the proxy

@quaint gyro do you have access to the machine running the engine?

so I can help you troubleshoot

could you ask your team to exec into the engine container and run this? curl https://repo.yarnpkg.com/tags?
^ if that doesn't work, then there might be an issue with the engine talking to the proxy server

[quaint gyro]

Argh! unfortunately no I don't have acces to it.

Ok I'll ask him 👍

[celest kernel]

thx 🙏

[quaint gyro]

otherwise, I was looking at Yarn env Vars and on their docs I found this:
https://yarnpkg.com/configuration/yarnrc

Finally, note that most settings can also be defined through environment variables (at least for the simpler ones; arrays and objects aren't supported yet). To do this, just prefix the names and write them in snake case: YARN_CACHE_FOLDER will set the cache folder (such values will overwrite any that might have been defined in the RC files - use them sparingly).

So if we set this config at the dagger engine level using env var just below proxy settings it should work I think

[celest kernel]

yes, just found the same. So we should be able to achieve what we need

first thing is to verify that the HTTP_PROXY is working correctly

[quaint gyro]

ok I'll tell you as soon as I have the curl command return!

[regal solar]

No we do not sadly 😦

[quaint gyro]

FYI I switched from feature issue to documentation issue, because it seems like every package managers handles custom configurations with environment variables:
https://github.com/dagger/dagger/issues/8056

I tried to fill somes information about Yarn, NPM, PIP.

May be it could great to have example of custom package manager handling for each one of them? Don't know whats the best way to help people with that tbh.

[quaint gyro]

So here is more infos:
we don't have curl and my colleagues thought that the wget command doesn't take proxy:
On the Worker node kube where dagger engine is running we can do this:

curl https://repo.yarnpkg.com/tags
{
  "latest": {
    "stable": "4.3.1",
    "canary": "4.3.1"
  },
  "aliases": {
    "latest": "4.3.1",
    "stable": "4.3.1",
    "canary": "4.3.1"
  },
  "tags": [
    "4.3.1",
    "4.3.0",
    "4.2.2",
...

From the pod it doesnt work but we think its related to wget.

If I understand well, he tried with wget from the dagger engine container with proxy env vars in lower case and we got that:

/ # wget https://repo.yarnpkg.com/tags
Connecting to proxy.foorbar-company.io:8080 (172.27.1.2:8080)
wget: server returned error: HTTP/1.1 500 handshakefailed

but it failed with google too so he doubt that the issue come from us:

/ # wget https://google.fr
Connecting to proxy.foorbar-company.io:8080 (172.27.1.2:8080)
wget: server returned error: HTTP/1.1 500 handshakefailed

From the worker node with curl, everything worked.

And finally the curl command from a pod with same proxy settings, in the same namespace than the Dagger Engine pod: it works.
His thoughts his that it maybe work inner Dagger Engine pod at system level if the pod had curl command, just that the Dagger's command doesn't have the proxy config with env vars.

[celest kernel]

we don't have curl and my colleagues thought that the wget command doesn't take proxy:

in the engine container they can run apk add curl to install curl

curl https://repo.yarnpkg.com/tags without manually specifying any proxy configuration should work since the engine should already have the HTTP_PROXY variables set

cc @quaint gyro

[quaint gyro]

he will try that tomorrow, I'll keep you posted

[quaint gyro]

Hi, so we try this in parallel, adding yarn/NPM var env like this:

          env:
            - name: HTTP_PROXY
              value: 'http://proxy.foobar-company.io:8080'
            - name: HTTPS_PROXY
              value: 'http://proxy.foobar-company.io:8080'
            - name: NPM_CONFIG_CAFILE
              value: /usr/local/share/ca-certificates/foobar-company-ca-cert
            - name: NPM_CONFIG_REGISTRY
              value: http://nexus.foobar-company.io/content/groups/public-npm-releases/
            - name: NPM_CONFIG_HTTPS_PROXY
              value: http://proxy.foobar-company.io:8080
            - name: NPM_CONFIG_NO_PROXY
              value: "*.foobar-company.io"
            - name: YARN_ENABLE_STRICT_SSL
              value: "false"
            - name: YARN_HTTPS_CA_FILE_PATH
              value: /usr/local/share/ca-certificates/foobar-company-ca-cert
            - name: YARN_NPM_REGISTRY_SERVER
              value: "http://nexus.foobar-company.io/content/groups/public-npm-releases/"
            - name: YARN_HTTP_PROXY
              value: http://proxy.foobar-company.io:8080
            - name: YARN_HTTPS_PROXY
              value: http://proxy.foobar-company.io:8080

It doesnt change anything, we always have a Request Error: getaddrinfo ENOTFOUND repo.yarnpkg.com error.

[quaint gyro]

Hi@celest kernel, fyi the command curl https://repo.yarnpkg.com/tags works on the dagger engine

My colleague think that maybe the generated job by dagger doesnt have proxy in it.

Whats the process of creating a dagger job exactly?
dagger cli use dagger-engine to:

• start a container
• install typescript sdk in it
• and run CI pipeline from it which call dagger engine to crreate job's container?

[celest kernel]

@quaint gyro what Dagger version are you using? (both server and client)

[quaint gyro]

the version for server and client are 0.12.1

[celest kernel]

@quaint gyro if you run dagger init --sdk typescript on a new empty folder. Does that work?

@quaint gyro by any chance do you have 20m to do a troubleshooting session together in #911305510882513037?

[quaint gyro]

nope, i got this:

$ dagger init --sdk=typescript

✔ connect 0.4s
[...]
✘ ModuleSource.asModule(engineVersion: "latest"): Module! 13.1s
! failed to create module: select: failed to update codegen and runtime: failed to generate code: failed to call sdk module codegen: select: call function "Codegen": process "/runtime" did not complete successfully: exit code: 2
Error: failed to generate code: input: moduleSource.withContextDirectory.withName.withSDK.withSourceSubpath.asModule resolve: failed to create module: select: failed to update codegen and runtime: failed to generate code: failed to call sdk module codegen: select: call function "Codegen": process "/runtime" did not complete successfully: exit code: 2

Stdout:
invoke: failed to create codegen base: could not list rootfs entries: input: container.from.withoutEntrypoint.withExec.withEnvVariable.withMountedCache.withExec.withWorkdir.withDirectory.withExec.withExec.withExec.directory resolve: process "yarn set version stable" did not complete successfully: exit code: 1

Stdout:
Request Error: getaddrinfo ENOTFOUND repo.yarnpkg.com
    at ClientRequest.<anonymous> (/root/.cache/node/corepack/v1/yarn/4.3.1/yarn.js:147:14258)
[...]
    at GetAddrInfoReqWrap.onlookupall [as oncomplete] (node:dns:120:26)
info Visit https://yarnpkg.com/en/docs/cli/policies for documentation about this command.
Stderr:
! Corepack is about to download https://registry.yarnpkg.com/yarn/-/yarn-1.22.22.tgz
! The local project doesn't define a 'packageManager' field. Corepack will now add one referencing yarn@1.22.22+sha512.a6b[...].
! For more details about this field, consult the documentation at https://nodejs.org/api/packages.html#packagemanager

! Corepack is about to download https://repo.yarnpkg.com/4.3.1/packages/yarnpkg-cli/bin/yarn.js
error Command failed.
Exit code: 1
Command: /usr/local/bin/node
Arguments: /usr/local/lib/node_modules/corepack/dist/corepack.js yarn@stable set version stable
Directory: /src
Output:

[celest kernel]

@quaint gyro by any chance do you have 20m to do a troubleshooting session together in #911305510882513037?

[quaint gyro]

I asked my colleague since he is the owner of the remote dagger engine and I dont have access to it

[celest kernel]

I don't think we need to change anything on the engine as I'd like to validate something on your side as well

[quaint gyro]

oh ok i will be available in 10min

[celest kernel]

at the same time, could you ask your colleague to run these two commands in the engine container and share the output please?

curl -v https://repo.yarnpkg.com/tags

NO_PROXY=repo.yarnpkg.com curl -v https://repo.yarnpkg.com/tags

[celest kernel]

thx 🙏 let me know. I'll be around

[quaint gyro]

im here, but dont know if it works correctly maybe because of the firewall

I can send you a Google meet in dm

[quaint gyro]

here is the output of the fist command:

and the output of the second:

NO_PROXY=repo.yarnpkg.com curl -v https://repo.yarnpkg.com/tags
* Uses proxy env variable NO_PROXY == 'repo.yarnpkg.com'
* Could not resolve host: repo.yarnpkg.com
* shutting down connection #0
curl: (6) Could not resolve host: repo.yarnpkg.com

[celest kernel]

ok, this is consistent with what we're seeing then

[quaint gyro]

Ok so @celest kernel we just tried that on dagger engine:

          env:
            - name: HTTP_PROXY
              value: 'http://proxy.foobar-company.io:8080'
            - name: HTTPS_PROXY
              value: 'http://proxy.foobar-company.io:8080'
            - name: _DAGGER_ENGINE_SYSTEMENV_NPM_CONFIG_CAFILE
              value: /usr/local/share/ca-certificates/foobar-company-ca-cert
            - name: _DAGGER_ENGINE_SYSTEMENV_NPM_CONFIG_REGISTRY
              value: http://nexus.dtni.foobar-company.io/content/groups/public-npm-releases/
            - name: _DAGGER_ENGINE_SYSTEMENV_NPM_CONFIG_HTTPS_PROXY
              value: http://proxy.foobar-company.io:8080
            - name: _DAGGER_ENGINE_SYSTEMENV_NPM_CONFIG_NO_PROXY
              value: "*.foobar-company.io"
            - name: _DAGGER_ENGINE_SYSTEMENV_YARN_ENABLE_STRICT_SSL
              value: "false"
            - name: _DAGGER_ENGINE_SYSTEMENV_YARN_HTTPS_CA_FILE_PATH
              value: /usr/local/share/ca-certificates/foobar-company-ca-cert
            - name: _DAGGER_ENGINE_SYSTEMENV_YARN_NPM_REGISTRY_SERVER
              value: "http://nexus.dtni.foobar-company.io/content/groups/public-npm-releases/"
            - name: _DAGGER_ENGINE_SYSTEMENV_YARN_HTTP_PROXY
              value: http://proxy.foobar-company.io:8080
            - name: _DAGGER_ENGINE_SYSTEMENV_YARN_HTTPS_PROXY
              value: http://proxy.foobar-company.io:8080

and it doesn't work

the test job:

import { connect, Client } from "@dagger.io/dagger"

// initialize Dagger client
connect(
    async (client: Client) => {
        // get Node image
        // get Node version
        const node = client.container().from("node:20-alpine@sha256:df01469346db2bf1cfc1f7261aeab86b2960efa840fe2bd46d83ff339f463665").withExec(["printenv"])

        // execute
        const version = await node.stdout()

        // print output
        console.log("Hello from Dagger and Node " + version)
    },
    { LogOutput: process.stderr },
)

here is the output:

> dagger run node --loader ts-node/esm ./build.mts
Setup tracing at https://dagger.cloud/traces/setup. To hide: export STOPIT=1

✔ connect 0.3s
✔ version: String! 0.0s
✔ container: Container! 0.0s
✔ Container.from(address: "node:20-alpine@sha256:df01469346db2bf1cfc1f7261aeab86b2960efa840fe2bd46d83ff339f463665"): Container! 0.0s
✔ Container.withExec(args: ["printenv"]): Container! 0.3s
✔ Container.stdout: String! 0.3s

(node:14276) ExperimentalWarning: `--experimental-loader` may be removed in the future; instead use `register()`:
--import 'data:text/javascript,import { register } from "node:module"; import { pathToFileURL } from "node:url"; register("ts-node/esm", pathToFileURL("./"));'
(Use `node --trace-warnings ...` to show where the warning was created)
(node:14276) [DEP0180] DeprecationWarning: fs.Stats constructor is deprecated.
(Use `node --trace-deprecation ...` to show where the warning was created)
(node:14276) [DEP0040] DeprecationWarning: The `punycode` module is deprecated. Please use a userland alternative instead.
Hello from Dagger and Node PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
NODE_VERSION=20.15.0
YARN_VERSION=1.22.22
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:44909
OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://127.0.0.1:44909/v1/traces
OTEL_EXPORTER_OTLP_TRACES_LIVE=1
OTEL_EXPORTER_OTLP_LOGS_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_LOGS_ENDPOINT=http://127.0.0.1:44909/v1/logs
OTEL_EXPORTER_OTLP_METRICS_PROTOCOL=http/protobuf
OTEL_EXPORTER_OTLP_METRICS_ENDPOINT=http://127.0.0.1:44909/v1/metrics
TRACEPARENT=00-725eb3aa2b73d5a856b314c5c9b26dcf-c06ebeeafd3711e8-01
HTTP_PROXY=http://proxy.foobar-company.io:8080
http_proxy=http://proxy.foobar-company.io:8080
HTTPS_PROXY=http://proxy.foobar-company.io:8080
https_proxy=http://proxy.foobar-company.io:8080
HOME=/root

[celest kernel]

can you try the dagger init --sdk typescript again in a new folder?

[quaint gyro]

yes i ll test that

otherwise, i give somes thoughts here in my issue because I don't know what could/should be the best way in a corporate context with proxies and internal registries:
https://github.com/dagger/dagger/issues/8056#issuecomment-2258602415

[quaint gyro]

gives me the Request Error: getaddrinfo ENOTFOUND repo.yarnpkg.com error too

[celest kernel]

@quaint gyro let me try if this works for me. One sec

we're very close 🙏

[celest kernel]

oh, nvm.. seems like it's not going to work. I was just checking at the code and seems like the _DAGGER_ENGINE_SYSTEMENV_* needs to be explicitly set and doesn't work for any random value https://github.com/dagger/dagger/blob/8233e147bc7ca08e0dc5667ea097b26a7c68d859/core/schema/sdk.go#L610

cc @muted robin IIUC this requires a new release? Wondering the reason of not making the SYSTEMENV thing open so users can decide what variables to pass to all the containers spawned by the engine

@quaint gyro could you try one last thing please? can you tell your ops team to also set the ALL_PROXY variable in the engine to see if that makes it work by any chance?

[muted robin]

That env var got added to unblock Go SDK users who needed GOPROXY, but it's more of a just a patch than a good long term approach.

The problem is we don't necessarily want to transparently pass these application specific env vars to every single container (unlike e.g. HTTP_PROXY/etc.). We'd start polluting every single container with tons of env vars that are meant for just one or a few containers. So right now the go sdk container just gets special cased to inherit that env for GOPROXY.

For the shorter term, we can keep playing whackamole and add one-off support for other envs like these ones for npm/yarn/etc. on the typescript SDK. But in the longer term we need to find some clean way of generalizing this. My initial attempts to design something were not great and ended up feeling like the equivalent of the windows registry, which is why we went with the quick hack for now.

See "Generalized custom proxy support" here: https://github.com/dagger/dagger/issues/6599

[celest kernel]

Gotcha, thx!

in this YARN / NPM case it's a bit more tricky since there are multiple NPM_* and YARN_* variables that could be set. @muted robin would calling os.Environ() around here https://github.com/dagger/dagger/blob/8233e147bc7ca08e0dc5667ea097b26a7c68d859/core/schema/sdk.go#L90-L93 be ok so we can scan and pass different SYSTEMENV variables to each runtime container accordingly?

[quaint gyro]

I think we will use the withEnv method, since its code we could have like a standard creation of job container for some uses case and reuse them.

rather than set a lots of vars for job which don't need it

[muted robin]

The way the Go SDK gets GOPROXY now is actually from here: https://github.com/dagger/dagger/blob/8233e147bc7ca08e0dc5667ea097b26a7c68d859/core/schema/sdk.go#L609-L617

The equivalent for other SDKs would be to make that same __withSystemEnvVariable call on this container: https://github.com/dagger/dagger/blob/8233e147bc7ca08e0dc5667ea097b26a7c68d859/core/schema/sdk.go#L245-L260

The gotcha is that it would only result in those env vars being set on the typescript runtime module: https://github.com/dagger/dagger/blob/8233e147bc7ca08e0dc5667ea097b26a7c68d859/sdk/typescript/runtime/main.go

So you'd need to grab the env vars in that code too and pass them anywhere they are needed (e.g. https://github.com/dagger/dagger/blob/8233e147bc7ca08e0dc5667ea097b26a7c68d859/sdk/typescript/runtime/main.go#L194-L204)