velvet pier
Hello,
Starting today, I have encountered numerous errors related to the title mentioned above, specifically within our staging environment. This issue seems to be linked to Redis, as other runs not involving the Redis database are functioning without any problems.
This is a critical issue for me. Could you please provide more detailed information about the error? Any deeper log insights you can share would be highly appreciated.
Thank you.
velvet pier
Using the same database in my dev environment does not reproduce the issue
I attempted to set up and use a new database, to figure out if it was something related to that specific DB, but it did not solve the problem.
The production environment uses a different database and is functioning correctly.
I recently upgraded to version 37.
The problem began around 14:00 UTC+2.
compact sentinel
Has prod been upgraded to beta.37 as well? How do you connect to redis?
velvet pier
Hi Nick, nope prod is still on 36. I upgraded to the 37 about 1 hour ago in staging.
I connect to redis using the upstash sdk
https://github.com/upstash/redis-js
compact sentinel
Looking into it. Is it okay to replay a few of those failed runs?
coral rose
I noticed that this run does actually have a redis error on the attempt: https://cloud.trigger.dev/orgs/test-trigger-netlify-1acc/projects/v3/defer-migration-G1Xp/runs/run_7od39d8vzn5sy8pj3dxgu?span=2c93283a1bd72cb2
If you click on the attempt you can see an error about "DB connection refused", and also for some reason there is a massive delay between Attempt 1 and Attempt 2, and then it fails with heartbeat timeout
velvet pier
thanks @coral rose , didn't notice this run in particual, but yeah i assumed there was some connection issue
velvet pier
Looking into it. Is it okay to replay a few of those failed runs?
sure Nick, no problem. The staging env is pretty much safe
compact sentinel
Okay, this is a replayed run https://cloud.trigger.dev/orgs/test-trigger-netlify-1acc/projects/v3/defer-migration-G1Xp/runs/run_h8opaxi1jdiwnxz58ozi2
There's no additional logs on our end. It seems to just sit there after those logs. We can see heartbeats so it's still "executing", but I can't tell you what exactly it's up to. It's not waiting on anything from our end.
velvet pier
uhm, at that point it should start retriving the data from the db. I will run the same payload in dev, if that could help you investigate furter
compact sentinel
That uses the same connection details?
velvet pier
exact same
compact sentinel
I did manage to get some additional logs from when that replayed run just failed. There's a memory leak somewhere.
I wonder if picking a higher RAM setting in the machine config would help. As far as I know we didn't change anything that would cause this in the last release.
Did you upgrade the upstash package recently by any chance?
velvet pier
uhm i could defenitly try
Also that you mentioned this, today around the time i tweaked some task in order to have more cpu and ram. This happend because i started to run some Heaver files from AWS bucket (where less than 1MB, now the are around 100MB)
compact sentinel
Let's hope it's just that, pick some beefier specs and try again 🤞
velvet pier
can i put them in the default config for now?
just as a quick try
look like i can't
compact sentinel
Oh you mean not on the task directly but the config file?
velvet pier
yeah, because i have like almost 100 task, and for a quick test that whould be faster to just put them into the configs
I put them in the task we where looking at togheter. Will be deployed in a few minutes
I think i found the issue @compact sentinel
If i deploy from the CLI, it restarts working properly.
Will retry to deploy with the github action, to check if that actually is the case
yep, looks like that the problem:
this run is a deployed version from github actions
https://cloud.trigger.dev/orgs/test-trigger-netlify-1acc/projects/v3/defer-migration-G1Xp/runs/run_kpp4fmpysgyoaam4q2464?span=161ec62d00b1d6f3
this one is from the cli intested:
https://cloud.trigger.dev/orgs/test-trigger-netlify-1acc/projects/v3/defer-migration-G1Xp/runs/run_yknap9vuvqt18n78g2jer?span=8cef6e69f483ac7e
same code, no changes
compact sentinel
Now I'm really confused
Is the CLI used in the github action on the same version? You haven't passed a flag to bypass version checks?
velvet pier
nope, i do not pin the version on porpouse, so i need to constantly upgrade the sdk
it is pretty much the same you provide in the docs
name: Deploy to Trigger.dev
on:
push:
branches:
- main
- production
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Use Node.js 20.x
uses: actions/setup-node@v4
with:
node-version: '20.x'
- name: Install pnpm
run: npm install -g pnpm
- name: Install dependencies
run: pnpm i
- name: Build
run: pnpm build
- name: 🚀 Deploy Trigger.dev
env:
TRIGGER_ACCESS_TOKEN: ${{ secrets.TRIGGER_ACCESS_TOKEN }}
run: |
if [ "${{ github.event.ref }}" = "refs/heads/main" ]; then
pnpm dlx trigger.dev@beta deploy --env staging --skip-typecheck
elif [ "${{ github.event.ref }}" = "refs/heads/production" ]; then
pnpm dlx trigger.dev@beta deploy --skip-typecheck
else
echo "Branch not supported for deployment"
exit 1
fi
Just with a twist so that i can deploy in staging/prod based on the branch
quick pollen
Just a note for tomorrow @coral rose that we should allow people to specify their default machine in the trigger.config file like @velvet pier wants. It makes sense to do this in your branch I think.
velvet pier
Thanks @quick pollen , that would be nice to have.
For @compact sentinel , i did some pnpm upgrade magic, cause we weren't upgrading the repo for like a month.
It looks like the action resumed to deploy to staging without that weird behavior when using redis. but i still wonder why it worked without issues nor upgade untill yesterday
compact sentinel
Thanks for the update! Glad that's resolved. 🙏