#Question about Bot Scaling with Job Queues

27 messages · Page 1 of 1 (latest)

ornate wasp
#

Hey everyone,
I'm building a Discord bot and want to design it for high scalability from the start – handling many guilds, no arbitrary limits, and graceful recovery when things crash.

Tech stack:
• TypeScript
• PostgreSQL + Drizzle ORM
• BullMQ (Redis) for job queues

My main concern: plugins register background jobs/tasks (reminders, scheduled actions, cron-like stuff, per-guild workloads). If the bot process crashes or restarts, I want:

No jobs lost
In-flight jobs resumed or retried cleanly
The system to scale horizontally across workers as guild count grows

How would you architect this? Specifically:
– Best practices for BullMQ in a multi-worker / multi-guild setup?
– Where should job state live – Redis only, or mirrored in Postgres for durability?
– How do you handle plugin-registered jobs so they auto-rehydrate after a crash?
– Any pitfalls when scaling past a few hundred / thousand guilds?
Appreciate any input, architecture diagrams, or battle-tested patterns! 🙏

tacit sundialBOT
ornate wasp
#

Question about Bot Scaling with Job Queues

pure lake
#

Pretty sure this entire question falls outside of the scope of discord.js

ornate wasp
pure lake
#

Well, we have #1081585952654360687, but i wouldn't expect much traffic there

#

Especially for questions like this

ornate wasp
#

sad :>

#

Can i have ur personal opinion about that then? 🌚

pure lake
#

No idea

#

Given how unsure how you'd want to tie d.js into it

ornate wasp
pure lake
#

Use to do what

#

Call the discord api endpoints? Use dedicated discord.js methods

#

Or don't use discord.js as a whole

#

Use /core directly instead

ornate wasp
#

I use for my Gateway-Process -> Client
Stuff for:
Receive slash commands (interaction.reply, deferReply)
Process button clicks
Gateway events (messageCreate, guildMemberAdd, voice-state-changes)
Cache queries (guild.members.cache, channel.permissionOverwrites.cache)

And for my Worker Process -> REST
Stuff for:
Cron jobs (send reminders, birthday announcements, end giveaways)
Queue processor actions (del-msg, channel-unlock)
Social notifications (post/edit live stream embeds)

#

They work paralell to each other

inner sentinel
#

Then you'd probably want to use @discordjs/proxy to have a single endpoint that keeps track of ratelimits, else your separate processes won't be able to keep tabs of how many requests are left in the ratelimit bucket

ornate wasp
inner sentinel
#

That's the global request limit. There's also route specific dynamic limits and sublimits though

ornate wasp
inner sentinel
ornate wasp
thin aurora
viral edge
#

I like rabbitmq

thin aurora