#Started sharding: a lot of `unknown interactions`

106 messages · Page 1 of 1 (latest)

severe basalt
#

I'v seen a consitant increase in unknown interaction since I started shardig (was barely getting any when it wasnt sharded).
My bot works mainly through Dms, which by default are all handled by shard 0 (so there is no need to communicate between shards). I do still get unknown interaction from dms buttons tho (like 1 out of 20).
I dunno what I could be possibly doing wrong since the code barely changed from before sharding to now.
And the first thing I try to do asap is to defer such interaction anyway

ShardingManager:

const { ShardingManager, ShardClientUtil } = require('discord.js');
const manager = new ShardingManager('/home/ubuntu/discord/index.js', {
    token: token,
    totalShards: Number(totalShards),  // Total number of shards you want to run
    shardList: [0,1,2,3,4], 
    respawn: true,  // Automatically respawn shards if they crash
});

//manager.on('shardCreate', shard => console.log(`Launched shard ${shard.id}`));

manager.on('shardCreate', (shard) => {
    console.log(`Shard ${shard.id} created.`);
   
    // Listen for messages from this shard
    shard.on('message', (message) => {
        console.log(`Message from shard ${shard.id}:`);
        const targetShard = manager.shards.get(message.shardId);
        if (targetShard) {
            targetShard.send(message)
                .then(() => {
                    console.log(`Message successfully sent to shard ${message.shardId}.`);
                })
                .catch((error) => {
                    console.error(`Failed to send message to shard ${message.shardId}:`, error);
                });
        } else {
            console.error(`Shard ${message.shardId} not available.`);
        }
    });
});

manager.broadcast({ action: 'log', message: 'A global event occurred.' });
manager.spawn();
storm ospreyBOT
#
  • What's your exact discord.js npm list discord.js and node node -v version?
  • Not a discord.js issue? Check out #1081585952654360687.
  • Consider reading #how-to-get-help to improve your question!
  • Explain what exactly your issue is.
  • Post the full error stack trace, not just the top part!
  • Show your code!
  • Issue solved? Press the button!
severe basalt
#

on the index.js file:

async function handleButtonInteraction(interaction) {
   await interaction.deferReply()
   // getting error "uknown interaction 1 out of 20 times
}


client.on('interactionCreate', async interaction => {
   if (interaction.isButton()) {
            if (interaction.customId.startsWith("ANS")) {
                await handleButtonInteraction(interaction)
                return
            }
            //...
})

So it's not that im doing much before trying to defer asap.
Im using OVH, If it's the provider the issue, before sharding I was still with OVH in europe w/o much interaction issues, now I moved it to Canada. east coast.. which should be even closer to Discord api

#

the bot is in 2k servers, so there isnt much traffic
(I have 4 cores and 8GB Ram)

brittle swan
severe basalt
brittle swan
#

I‘m just curious why totalShards is a variable but the shardList is constant

serene wave
#

also curious as to what's the actual value of that totalShards

severe basalt
#

totalshard is 5
shardlist is the shards that should run on this machine, while in the future other shards will operate on separate machines
hence why the totalshard is in the file that I will just copy over to external vps

#

(im not yet doing that tho)

brittle swan
#

But in that case it sounds like the shardList would be the one thing you‘d want to differ per environment. not hardcoded in the code

severe basalt
#

isnt totalshards accross all machines, while shardlist is what is assigned to this specific machine?

serene wave
#

doesn't that mean you would want to give the shardlinst via env as well, so that you can just copy paste the same code across machines?

severe basalt
#

indeed that's for the future when I'll add external ones

brittle swan
#

Why are you running 5 shards for 2k servers?

severe basalt
#

So I dont have to worry for a long time

brittle swan
#

Especially considering you say yourself that most work will be on shard 0 either way you‘re basically slowing your bot down.

brittle swan
severe basalt
#

well is mainlny coz of the 2.5k limit per shard, since iv 4 cores and 8gb ram I'd imagine 5 shards is not that demanding?

#

As in, shard 0 handles dms, data should directly go to such shard w/o bouncing around, unless you saying this is what actually happen?

brittle swan
#

No, it isn’t. But you‘re limiting your shard 0 to less resources when you force the ShardingManager to spawn more shards

severe basalt
#

it's not that busy

brittle swan
#

Do the interactions actually fail on client side or do you only get unknown interaction errors while the user still gets replies?

#

Because why are you running so many instances of your ShardingManager?

severe basalt
#

well im consistently getting them every 20ish interactions, but never on my end when testing.
I tried lately

async function handleButtonInteraction(interaction) {
    let unknownInteraction = false
    try {
        await interaction.deferReply()
    } catch (error) {
        unknownInteraction = true
    }
    
    if (unknownInteraction) {
          interaction.channel.send({ content: "text"})
    } else {
          interaction.editReply({ content: "text" })
    }
}

but now im getting

/home/ubuntu/node_modules/@discordjs/rest/dist/index.js:727
      throw new DiscordAPIError(data, "code" in data ? data.code : data.error, status, method, url, requestData);
            ^

DiscordAPIError[50035]: Invalid Form Body
message_reference[REPLIES_CANNOT_REPLY_TO_SYSTEM_MESSAGE]: Cannot reply to a system message
    at handleErrors (/home/ubuntu/node_modules/@discordjs/rest/dist/index.js:727:13)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async SequentialHandler.runRequest (/home/ubuntu/node_modules/@discordjs/rest/dist/index.js:1128:23)
    at async SequentialHandler.queueRequest (/home/ubuntu/node_modules/@discordjs/rest/dist/index.js:959:14)
    at async _REST.request (/home/ubuntu/node_modules/@discordjs/rest/dist/index.js:1272:22)
    at async DMChannel.send (/home/ubuntu/node_modules/discord.js/src/structures/interfaces/TextBasedChannel.js:182:15) {
  requestBody: {
    files: [],
    json: {
      content: 'Answer must contain characters',
      tts: false,
      nonce: undefined,
...

which is actually an error that reboots the shard - even tho everything is in a try catch

#

Because why are you running so many instances of your ShardingManager?

Sure I could change it, but shouldnt 4 cores and 8bg be more than enough to handle 5 shards?
Even more so now that 2k servers are spread among 5 shards

#

No, it isn’t. But you‘re limiting your shard 0 to less resources when you force the ShardingManager to spawn more shards
Sure, I mean when/if I'll have 200.000 servers, this main server will probably be just for shard 0 and worker threads I'd imagine, but im far from this scenario currently

brittle swan
severe basalt
#

Are you implying that, with this current server specs, having 5 shards is the cause of the unknown interaction?

brittle swan
#

No. I have no information what the cause is. Only theories. Finding the cause can only be done by you, the one actually running the bot. We can only hint at things you could/should look at.

#

If you never had any user get an „Application did not respond“ in discord itself then it could also be that you‘re simply running multiple instances of your shard 0

severe basalt
#

So it's not that common to get unknown interactions when switching to sharding?
In that case it could also me being unlucky on having been assigned by ovh a server that is actually busy lol, since it's not dedicated.

brittle swan
#

Did you actually read what I said?

severe basalt
#
 ps aux | grep node
ubuntu   3964007  0.3  0.8 21976812 69364 ?      Ssl  15:50   0:02 node /home/ubuntu/discord/shardingManager.js
ubuntu   3964017  1.2  2.2 22379496 178488 ?     Sl   15:50   0:08 /home/ubuntu/.nvm/versions/node/v20.10.0/bin/node /home/ubuntu/discord/index.js
ubuntu   3964048  0.9  1.8 22364612 149760 ?     Sl   15:50   0:06 /home/ubuntu/.nvm/versions/node/v20.10.0/bin/node /home/ubuntu/discord/index.js
ubuntu   3964068  1.0  2.0 22369348 160676 ?     Sl   15:50   0:07 /home/ubuntu/.nvm/versions/node/v20.10.0/bin/node /home/ubuntu/discord/index.js
ubuntu   3964089  0.9  1.8 22361976 150136 ?     Sl   15:50   0:06 /home/ubuntu/.nvm/versions/node/v20.10.0/bin/node /home/ubuntu/discord/index.js
ubuntu   3964110  0.9  2.1 22366124 167836 ?     Sl   15:50   0:06 /home/ubuntu/.nvm/versions/node/v20.10.0/bin/node /home/ubuntu/discord/index.js

looks good here

#
manager.on('shardCreate', (shard) => {
    console.log(`Shard ${shard.id} created.`);
client.on('ready', async () => {
    console.log("REBOOT: ", `Shard #${client.shard.ids} is ready! <<<<`);
 Shard 0 created.
 REBOOT:  Shard #0 is ready! <<<<
 Shard 1 created.
 REBOOT:  Shard #1 is ready! <<<<
 Shard 2 created.
 REBOOT:  Shard #2 is ready! <<<<
 Shard 3 created.
 REBOOT:  Shard #3 is ready! <<<<
 Shard 4 created.
 REBOOT:  Shard #4 is ready! <<<<
severe basalt
#
    sweepers: {
        guildMembers: {
            interval: 3_600, 
            lifetime: 24 * 60 * 60, 
            filter: () => user => user.id === user.client.user.id,
        },
        threads: {
            interval: 86_600, 
            lifetime: 7 * 24 * 60 * 60, 
        },
    },
    makeCache: Options.cacheWithLimits({
        ...Options.DefaultMakeCacheSettings,
       needed
        GuildMemberManager: {
            maxSize: 10,
            keepOverLimit: (member) => member.id === client.user.id,
        },
        ThreadManager: 30, 
    }),

could these sweepers be the cause?

serene wave
#

interactions aren't cacheable

#

and whoever used them gets sent to you alongside them

#

and even if you're fetching the members (doubt, cause that's not a concept in DMs, and neither are threads)... that has like, 0 to do with you deferring interaction before you do anything anyway

#

d.js doesn't fetch anything on its own, you have to tell it to. it's like that for it fetching gateway address from the login(), it's like that for the fetchReply param in some interaction response options (which is now deprecated because discord added native way to receive anything back from that with the withResponse option).

severe basalt
# serene wave interactions aren't cacheable

ya was worried of some under the hood things that may be causing these unknown interactions..

btw interaction.createdTimestamp is the same time which discord expects a reply within the 3 secs span?

serene wave
#

no idea

severe basalt
#
async function handleButton(interaction) {
 console.log(interaction.id, "INT ID")
    const msElapsed = Date.now() - interaction.createdTimestamp;
    console.log(msElapsed, "DELAY")
    let unknownInteraction = false
    try {
        await interaction.deferReply()
    } catch (error) {
        unknownInteraction = true
        console.log(client.shard.ids, 'CANT DEFER BTN ANS ', msElapsed, "DELAY")
    }
76 DELAY
[ 0 ] CANT DEFER BTN ANS  76 DELAY

im also not seeing interactions ids duplicates

#

now that i reduced it to 2 shards im may be seeing less frequent errors

#

for a 4 vcores 8bg ram what is the suggested amount of shards such machine can host?

serene wave
#

that depends on way too many things

#

you could probably squeeze a million servers there, you could make it not handle a single command

severe basalt
#

well im really lost
my bot is not that busy atm
dunno if setting the process to higher priority would fix this weird unknown interaction

severe basalt
#
 ConnectTimeoutError: Connect Timeout Error
   at onConnectTimeout (/home/ubuntu/node_modules/undici/lib/core/connect.js:190:24)
   at /home/ubuntu/node_modules/undici/lib/core/connect.js:133:46
   at Immediate._onImmediate (/home/ubuntu/node_modules/undici/lib/core/connect.js:174:9)
   at process.processImmediate (node:internal/timers:478:21) {
   code: 'UND_ERR_CONNECT_TIMEOUT'
 }

well this is a new error I never seen before

#
manager.spawn({ delay: 5000 });

should I add a delay when spawning shards? Does discord prefer it?

serene wave
#

Discord requires it. One identify per 5 seconds. But given how it's a requirement and how you don't have to specify it, it's handled automatically for you

gilded jungleBOT
#

tag suggestion for @severe basalt:

Errors such as ECONNRESET, ConnectTimeoutError, SocketError: other side closed indicate a network issue

  • Free or shared hosting providers often come with significant limitations
  • Another indication of these issues are 'Unknown Interaction' errors.
brittle swan
#

That together with your unknown interaction issue makes it even clearer this is a network issue of your host

severe basalt
#
const msElapsed = Date.now() - interaction.createdTimestamp;
    console.log(msElapsed, "DELAY")
    try {
        await interaction.deferReply()
    } catch (error) {
        console.log(client.shard.ids, 'CANT DEFER BTN ANS ', msElapsed, "DELAY")
    }
36 DELAY
[ 0 ] CANT DEFER BTN ANS  36 DELAY
polar tundra
#

well you receive interactions via ws, but you reply to them via rest

#

so technically they can be entirely unrelated, and there can be issues with one but not the other

brittle swan
#

You also calculate the difference before even starting to send the defer. If you want accurate results you should calculate them in the catch

#

And even then it depends on your host‘s clock being in sync with discord’s clock

severe basalt
#

im usinv ovh canada
would changing dns to cloudflare be of any help?

serene wave
#

dns has practically nothing to do with connection to discord

#

only thing dns does is "convert" discord.com into one of cloudflare ip addresses

polar tundra
severe basalt
#

that is indeed the case

#
 try {
        await interaction.deferReply()
    } catch (error) {
        const msElapsed = Date.now() - interaction.createdTimestamp;
        console.log(client.shard.ids, 'CANT DEFER BTN ANS ', msElapsed, "DELAY")
    }
[ 0 ] CANT DEFER BTN ANS  5156 DELAY
[ 0 ] CANT DEFER BTN ANS  5108 DELAY
polar tundra
#

yeah your defer is taking way too long to reach

severe basalt
#

tho im getting it once every 30 other interactions that do work well

#

so im not sure what could it be

brittle swan
#

Something is blocking your request. Listen to rateLimited event and also make sure you don’t have any sync I/O calls clogging your event loop

#

What is it this bot does exactly?

severe basalt
#

processing applications through dms, but im getting this apparent delay not only from dms butons but from any guild as well

#
client.rest.on('rateLimited', (info) => {
    console.log(info)
})

?

#

will try to see tools that checks I/O but for sure htop shows that all is quiet
very quiet

#

only thing I can say is that when I had this normal bot on a 1vcore 2gb ram vps in germany I was barely getting any unknown interactions, now in canada (which should be closer to discord api) with 4c and 8gb I ran into these probelms 😄

severe basalt
#

running some debugging tools and all seems to be normal
Tho im wondering, if it was some sync ops clogging up the system, shouldnt I get delays when first receiving the interaction as well? - and not only when trying to defer them?

serene wave
#

receiving is not the same as sending

#

those are not only distinct processes, but also are invoked by different processes and sides, and in this case - as already mentioned - work over completely different protocols

#

internet in my building can suddenly drop upload. my download is all fine, no packet losses and absolutely full speed, but upload is a literal 0. handful of packets can make it outside. any speedtest i attempt will not work and error out on the upload part of it

severe basalt
#

right but if the cause of this were whatever internal long running sync op, I would be seeing issues on both send/ receive. Since this is not the case, im out of options beside external server provider issues

severe basalt
#
const fetch = require('node-fetch');

async function manuallyDefer(interaction) {
    const url = `https://discord.com/api/v10/interactions/${interaction.id}/${interaction.token}/callback`;

    const body = {
        type: 5 // DEFERRED_CHANNEL_MESSAGE_WITH_SOURCE
    };

    const start = Date.now();
    console.log(`[${interaction.id}] Sending raw deferReply at ${start}`);

    try {
        const res = await fetch(url, {
            method: 'POST',
            headers: {
                'Authorization': `Bot ${process.env.BOT_TOKEN}`,
                'Content-Type': 'application/json'
            },
            body: JSON.stringify(body)
        });

        const end = Date.now();
        console.log(`[${interaction.id}] Response from deferReply in ${end - start}ms`);

        if (!res.ok) {
            const errText = await res.text();
            console.error(`[${interaction.id}] Defer failed: ${res.status} - ${errText}`);
        }
    } catch (error) {
        console.error(`[${interaction.id}] Defer error:`, error);
    }
}

Should I try this to manually debug the issue

severe basalt
#

im noticing im getting unknown interaction on another bot im running as well, which is not sharded

brittle swan
#

Is that running on a different host?

severe basalt
brittle swan
#

Then it‘s the same network issue. Since they share the same network connection

#

How many other bots are you running on there?

severe basalt
#

i have 1Gbps, as the graph supposedly say im way below it

severe basalt
#

and 2 websites

#
 try {
        await interaction.deferReply()
    } catch (error) {
        const memoryUsage = process.memoryUsage();
        console.log(`Heap Used: ${(memoryUsage.heapUsed / 1024 / 1024).toFixed(2)} MB`);
        console.log(`Heap Total: ${(memoryUsage.heapTotal / 1024 / 1024).toFixed(2)} MB`);
Heap Used: 188.32 MB
Heap Total: 193.33 MB

is this a concen?

#

also:

setInterval(() => {
    const currentTime = process.hrtime(lastTime);
    const delay = currentTime[0] * 1e3 + currentTime[1] / 1e6; 

    if (delay > 1050) {  // Log if the delay exceeds 100 ms (adjust threshold as needed)
        console.log('Event Loop Delay:', delay, 'ms');
    }

    lastTime = process.hrtime();  
}, 1000);

im barely getting a log, if not only on reboot

severe basalt
#
const blocked = require('blocked-at');
blocked((time, stack) => {
  console.log(`Blocked for ${time}ms, operation started here:`, stack);
}, { threshold: 50 }); // Logs if event loop is blocked > 50ms

also rarely getting this, and not when getting unknwon interaction

serene wave
severe basalt
#

can promises be realistically the problem?

#

as in any async process

#

or
to have these issues, it should rather be sync ops issue

serene wave
#

have you tried with a different provider yet?

#

if promises would cause this, half the internet would claim that node is unusable

severe basalt
#

that's assuring 😄

severe basalt
severe basalt
#

btw sidenote, i tried pinging discord with traceroute
and it seems the connection is not direct as one might think
as in there actually wouldnt be much difference between having a bot in north america and europe

serene wave
#

cloudflare, yes