Payload Not Executing Exclusive Workflow Queued Jobs

[gilded hatch]

I had an issue today. Payload wasn't executing queued jobs of a exclusive workflow with supersedes: false.
Never happened this before, i checked the logs for some reasons it stopped executing this workflow meanwhile scheduler keeps scheduling the new jobs.

I checked vercel logs, everything was fine, no errors.
Then i deleted all the pending jobs from the same queue and it worked, payload started executing them.

Is it the expected behaviour or something was wrong with my code?
I was assuming payload would clear up the queue one by one in a specific order.

  concurrency: {
    exclusive: true,
    supersedes: false,

    key({ input, queue }) {
      return `queue#${queue}-gameSeason#${getDocumentId(input.gameSeason)}`;
    },
  },

[lime lily]

Original message from @leaden barn - Moved from #general message

New Community-Help Thread Created!

Help is on the way! To mark it as solved, use the /solve command. In the meantime, here are some existing threads that may help you:

Documentation:

• Queues - Executing jobs - Cron jobs
• Job Schedules - Defining schedules on Tasks or Workflows - Example schedule
• Job Schedules - Troubleshooting Schedules - Scheduled Jobs queued but not running
• Queues - Executing jobs - Endpoint - Query Parameters
• Jobs - Job Options - Cancelling Jobs

[amber oak]

Hey @leaden barn — this is a great question! Looks like there is a missing piece in the docs about how to address stale jobs.

The reason you're not seeing errors, but your jobs didn't execute is likely the Vercel function was killed mid-execution (timeout, cold start, deploy). So the job never got to set processing: false, leaving a ghost job permanently blocking the entire queue.

You could address that with something like this, on \a shorter schedule to ensure it catches if/before timeout

tasks: [
  {
    slug: 'cleanup-stuck-jobs',
    handler: async ({ req }) => {
      const staleThreshold = new Date(Date.now() - 10 * 60 * 1000) // 10 minutes ago
      await req.payload.update({
        collection: 'payload-jobs',
        where: {
          and: [
            { processing: { equals: true } },
            { updatedAt: { less_than: staleThreshold.toISOString() } },
          ],
        },
        data: { processing: false },
      })
      return { output: {} }
    },
  },
]

[leaden barn]

thanks sean for helping me out here. I'm pretty sure it has nothing to with vercel functions. I did checked the vercel cron logs multiple times to find out the root cause but couldn't find anything convincing enough from the infra layer.

• I didn't had any recent changes to the workflow implementation.
• The workflow was running fine for more than a month until the last night.

However, i did made a change which is upgrading the project deps recently, i have attached the diffs.

Even after updating the deps, the workflow ran nicely as i confirmed it from the job logs.

We can think about the work around but first i'm more interested in knowing if this is an expected behaviour.

I'm gonna try to re-produce it maybe this weekend.

@amber oak