Self-Hosted v3 in Kubernetes - Invalid Token error from Kubernetes Provider | Trigger.dev | Page 1

hallow pollen Jul 22, 2024, 6:09 AM

#

I am trying to deploy a Self-Hosted Trigger v3 in Kubernetes. I have successfully deployed the Web App (can access the dashboard) and Coordinator. Only the Kuberntes Provider is not working due to some error. Here are some possible related logs:

Web App

[Jul 22 2024 14:14:10 GMT+0800] trigger-v3-kapitol-prod-86b94bdc8c-kb9lw: Attemping to upgrade connection at url /socket.io/?EIO=4&transport=websocket with headers: {"x-trigger-provider-type":"kubernetes","sec-websocket-version":"13","sec-websocket-key":"X33tdG4T2/xYJb+gYnM0iw==","connection":"Upgrade","upgrade":"websocket","sec-websocket-extensions":"permessage-deflate; client_max_window_bits","host":"trigger-v3-kapitol-prod-service"}
[Jul 22 2024 14:14:10 GMT+0800] trigger-v3-kapitol-prod-86b94bdc8c-kb9lw: Socket.io client connected, upgrading their connection...
[Jul 22 2024 14:14:10 GMT+0800] trigger-v3-kapitol-prod-86b94bdc8c-kb9lw: {"socketId":"xwPQmlnWaIBltAAnAAAD","socketStage":"auth","timestamp":"2024-07-22T06:14:10.346Z","name":"provider","message":"invalid token","level":"error"}
[Jul 22 2024 14:14:10 GMT+0800] trigger-v3-kapitol-prod-86b94bdc8c-kb9lw: {"socketId":"sWsVHejGpjoosJMXAAAE","socketStage":"auth","timestamp":"2024-07-22T06:14:10.347Z","name":"shared-queue","message":"invalid token","level":"error"}
[Jul 22 2024 14:14:10 GMT+0800] trigger-v3-kapitol-prod-86b94bdc8c-kb9lw: {"pattern":"marqs:*sharedQueue","component":"marqs","operation":"rebalanceParentQueues","service":"marqs","timestamp":"2024-07-22T06:14:10.544Z","name":"webapp","message":"Streaming parent queues based on pattern","level":"debug"}

How do I fix the "invalid token" error here? Or what should be the values for the PLATFORM_SECRET which causes the error?

The current setup has the same values for the PLATFORM_SECRET in the web app and the PROVIDER_SECRET in the kubernetes-provider.

hallow pollen Jul 22, 2024, 6:31 AM

#

Kubernetes Provider Logs

[Jul 22 2024 14:22:51 GMT+0800] trigger-v3-provider-kapitol-prod-c48df47-pgj8p: [local] running in kubernetes mode
[Jul 22 2024 14:22:52 GMT+0800] trigger-v3-provider-kapitol-prod-c48df47-pgj8p: {"uri":"ws://trigger-v3-kapitol-prod-service:80/provider","timestamp":"2024-07-22T06:22:52.044Z","name":"provider","message":"new zod socket","level":"log"}
[Jul 22 2024 14:22:52 GMT+0800] trigger-v3-provider-kapitol-prod-c48df47-pgj8p: {"uri":"ws://trigger-v3-kapitol-prod-service:80/shared-queue","timestamp":"2024-07-22T06:22:52.241Z","name":"shared-queue","message":"new zod socket","level":"log"}
[Jul 22 2024 14:22:52 GMT+0800] trigger-v3-provider-kapitol-prod-c48df47-pgj8p: [TaskMonitor] Connected
[Jul 22 2024 14:22:52 GMT+0800] trigger-v3-provider-kapitol-prod-c48df47-pgj8p: [PodCleaner] Starting
[Jul 22 2024 14:22:52 GMT+0800] trigger-v3-provider-kapitol-prod-c48df47-pgj8p: [local] Uptime heartbeat is disabled, set UPTIME_HEARTBEAT_URL to enable.
[Jul 22 2024 14:22:52 GMT+0800] trigger-v3-provider-kapitol-prod-c48df47-pgj8p: [local] server listening on port 8080
[Jul 22 2024 14:22:53 GMT+0800] trigger-v3-provider-kapitol-prod-c48df47-pgj8p: file:///app/index.mjs:103682
[Jul 22 2024 14:22:53 GMT+0800] trigger-v3-provider-kapitol-prod-c48df47-pgj8p:                  reject(new apis_1.HttpError(response, body, response.statusCode));

agile fiber Jul 22, 2024, 12:19 PM

#

We don't currently have support for Kubernetes in self-hosting. Right now we provide docs and support for the Docker setup but not anything else.

hallow pollen Jul 23, 2024, 1:54 AM

#

Hey @agile fiber , just to confirm, does the kubernetes-provider exists for future Kubernetes support? Is it in the roadmap or it just for Trigger Cloud for now?

agile fiber Jul 23, 2024, 7:35 AM

#

It’s just for the cloud product at the moment although we’re actually going to be moving away from Kubernetes soon – it’s too slow to spin up workers (takes 4–5 seconds to start a run at scale)

hallow pollen Jul 23, 2024, 8:42 AM

#

agile fiber It’s just for the cloud product at the moment although we’re actually going to b...

Got this. We may need to stick on a Self-hosted v2 or Cloud v3 for the time being

agile fiber Jul 23, 2024, 8:43 AM

#

How many concurrent runs do you need to do?

hard walrus Sep 3, 2024, 10:09 PM

#

agile fiber It’s just for the cloud product at the moment although we’re actually going to b...

Hey @agile fiber, out of curiosity, what did you end up going for? A custom orchestrator + Firecracker or more of an alternative to Kubernetes like Nomad? I was looking at getting v3 working on Kubernetes, but if it's not going to be officially supported in the near future I might have to look for a different solution, since I wanted to self-host Trigger.dev

agile fiber Sep 4, 2024, 11:35 AM

#

We haven't shipped Firecracker yet. I think we're going to use Pulumi internally to deploy, manage and scale the cloud though.

One critical move we are making which will make the self-hosted experience much better is having the workers pull tasks from the queue. Right now it's push-based. That will make deploying and scaling self-hosted way better. We're working on this at the moment because we're going to use it as well.

#Self-Hosted v3 in Kubernetes - Invalid Token error from Kubernetes Provider