Unable to run Trigger.dev after the new tunnel update

[gritty plinth]

Getting this error:

⬆️ [trigger:dev] Service ready
   ▲ Next.js 14.0.4
   - Local:        http://localhost:3100

✔️ [trigger.dev] Detected TriggerClient id: caddie-next-8XQb
✔️ [trigger.dev] Found API Key in process runtime
- [trigger.dev] Looking for your trigger endpoint: http://localhost:3100/api/trigger
 ✓ Ready in 2.5s
 ○ Compiling /api/trigger ...
 ✓ Compiled /api/trigger in 3.1s (514 modules)
✔ [trigger.dev] Found your trigger endpoint: http://localhost:3100/api/trigger
- 🚇 Creating tunnel
✖ Ngrok failed to create a tunnel for port 3100.
connect ECONNREFUSED 127.0.0.1:4040
❌ [trigger:dev] Service exited unexpectedly

Any idea what thats about? I verified my env vars are correct. Command: npx @trigger.dev/cli@latest dev -p 3100

[languid inlet]

@gritty plinth are you self hosting the Trigger.dev platform? We added a new tunnel that doesn't have an aggressive rate limit. It falls back to Ngrok if you're self hosting though because we host the new tunnel.

[gritty plinth]

I am self hosting

This is the first time ive ran the dev server today so im not sure if its the rate limiter

[languid inlet]

Yeah it's not that, I was just trying to figure out why it was still using ngrok

For now try using the version before we added the new tunnel:

npx @trigger.dev/[email protected] dev -p 3100

I'll create an issue and we can figure out how we broke backwards compatibility, I didn't work on this but usually we make everything backward and forward compatible. Sorry about this.

Issue here: https://github.com/triggerdotdev/trigger.dev/issues/803

[languid inlet]

@gritty plinth actually there is one other possibility, I think that error also gets shown when the TRIGGER_API_URL and TRIGGER_API_KEY aren't set correctly.

[gritty plinth]

Let me double check, I did printenv | grep "TRIGGER" and it looked right at the time I posted this but ill double check

[gritty plinth]

Hmmm odd, my API keys are correct, I just verified

but even 2.3.5 doesnt work anymore

[languid inlet]

⬆️ [trigger:dev] Service ready
   ▲ Next.js 14.0.4
   - Local:        http://localhost:3100

✔️ [trigger.dev] Detected TriggerClient id: caddie-next-8XQb
✔️ [trigger.dev] Found API Key in process runtime
- [trigger.dev] Looking for your trigger endpoint: http://localhost:3100/api/trigger
 ✓ Ready in 2.5s
 ○ Compiling /api/trigger ...
 ✓ Compiled /api/trigger in 3.1s (514 modules)
✔ [trigger.dev] Found your trigger endpoint: http://localhost:3100/api/trigger
- 🚇 Creating tunnel
✖ Ngrok failed to create a tunnel for port 3100.
connect ECONNREFUSED 127.0.0.1:4040
❌ [trigger:dev] Service exited unexpectedly

Looking at your original error again 👆.

Ngrok is responding with connect ECONNREFUSED 127.0.0.1:4040. It looks like Ngrok is trying to connect to your 4040 port as part of some kind of handshake, given your port is 3100 for your site.

There's an ngrok issue here about this issue which might be related: https://github.com/bubenshchykov/ngrok/issues/138

It looks like you can fix it by hitting http://localhost:4040 on your machine, probably just via a browser. Then trying again.

[devout cedar]

@languid inlet also self hosting and hitting the same issue ^

the workaround with hitting http://localhost:4040 doesn't seem to work for us

[devout cedar]

Also tried using CLI versions 2.3.5 and 2.3.2 with similar results

[languid inlet]

What version of the platform are you on? It seems like we might have introduced an issue with ngrok in the latest release, it isn’t used on the cloud anymore

[devout cedar]

We've been self hosting v2.3.2

[languid inlet]

Sorry I mean the docker image version. The latest is v.docker.2.2.25

[gritty plinth]

Is there a way to check in the UI? I just push the latest tag so I’m not sure how to check

[gritty plinth]

So i tried to create a fresh server with the latest trigger and had issues with the graphile_worker table not existing

and on my old instance, jobs stopped running

[gritty plinth]

Fixed it 😅 , had to set NODE_TLS_REJECT_UNAUTHORIZED when using digital ocean managed databases because the postgres cert is self signed

might be good to document somewhere or throw a better error message

[languid inlet]

Is there anything else for Digital Ocean that didn't match our guides? https://trigger.dev/docs/documentation/guides/self-hosting

We can add Digital Ocean as a guide here, with that note on it.

[gritty plinth]

Everything is the exact same as fly except for needing to have your own dockerhub image (perhaps fixed in the future) and needing to set NODE_TLS_REJECT_UNAUTHORIZED='0'. Without that env var, you get the following error from graphile:

[2023-12-22 10:08:56] Error: self-signed certificate in certificate chain
[2023-12-22 10:08:56]     at TLSSocket.onConnectSecure (node:_tls_wrap:1600:34)
[2023-12-22 10:08:56]     at TLSSocket.emit (node:events:517:28)
[2023-12-22 10:08:56]     at TLSSocket._finishInit (node:_tls_wrap:1017:8)
[2023-12-22 10:08:56]     at ssl.onhandshakedone (node:_tls_wrap:803:12) {
[2023-12-22 10:08:56]   code: 'SELF_SIGNED_CERT_IN_CHAIN'
[2023-12-22 10:08:56] }

There is also no indication that the error is from graphile (would be nice to have a initializing graphile log right before)

Sorry I digressed from the orignal thread

Where does trigger get the ngrok cli from? (for local dev)

[languid inlet]

We should definitely have a clearer log for that. Did you see: "Initializing worker queue with options"? We can add "graphile" to that. I'm wondering if it failed before that

[languid inlet]

You can use the ngrok npm package (https://www.npmjs.com/package/ngrok) to create a tunnel for free just using an SDK call. It has some fairly aggressive rate limits which can be an issue for some people who are doing a ton of runs locally during development.

When you're using the CLI you can always pass in your own tunnel:

npx @trigger.dev/cli@latest dev -t https://whatever.mytunnel.com/

[gritty plinth]

Ah yeah that log is there but its a super long json so I didnt notice it, and it was followed by SqsEventConsumer is disabled, No PostHog API key, app ready: http://localhost:3030, and prisma:info Starting a postgresql pool with 10 connections. before the error so I assumed it had nothing to do with that

Gotcha, I wanted to see if I can help debug whats going on with the cli to see what the connect ECONNREFUSED 127.0.0.1:4040 error was, I assumed you guys used the npm ngrok package but running npx ngrok asks me to install it so im not sure

One interesting fact, ngrok on my computer actually routes to the cloudflared command, idk if that makes a difference

[languid inlet]

I think because we're using their SDK that wouldn't be the problem, but I guess it's possible.

I've added "graphile" to some of the logs when the worker starts up: https://github.com/triggerdotdev/trigger.dev/commit/aaea064208aee606b8e29a14e5dc2327b9ec0bf1

[gritty plinth]

Fixed it, I had to signup for an ngrok account and set the auth token and it works now. Might be worth adding instructions here: https://trigger.dev/docs/documentation/guides/tunneling-platform#start-ngrok

Perhaps they have a "start a server locally 10 times before requiring authentication" rule or something

[languid inlet]

Ok that's interesting, this must be a new restriction. It doesn't surprise me to be honest because they do give away a lot for free. All the other free tunneling services don't allow API traffic