discordjs/ws big bot memes (old) | discord.js - imagine an app | Page 3

sullen snow Apr 13, 2023, 1:19 PM

#

@stable hatch so far its fine to me

#

since the ws actually heartbeats

#

this is on 50 shards per cluster with 1x concurrency

#

so the non identified shards didnt dc on this time frame

stable hatch Apr 13, 2023, 1:21 PM

#

oh wait

#

do heartbeats keep it alive?

#

@dusty dove i'm testing my hypothesis by spawning 255 shards

#

uh

#

wat

#

[251] Identifying
    shard id: 251
    shard count: 256
    intents: 0
    compression: none
[251] Waiting for event ready for 15000ms
[251] Ready
Connected
[251] Identifying
    shard id: 251
    shard count: 256
    intents: 0
    compression: none
[251] Waiting for event ready for 15000ms
[251] More than one auth payload was sent.
[251] Destroying shard
    Reason: none
    Code: 4005
    Recover: Reconnect```

dusty dove Apr 13, 2023, 1:23 PM

#

WAYTOODANK

stable hatch Apr 13, 2023, 1:23 PM

#

only shard that had this issue somehow

dusty dove Apr 13, 2023, 1:23 PM

#

no idea what that's all about

stable hatch Apr 13, 2023, 1:23 PM

#

on your PR

dusty dove Apr 13, 2023, 1:23 PM

#

stable hatch do heartbeats keep it alive?

yeah they should

stable hatch Apr 13, 2023, 1:23 PM

#

yeah me neither, somehow it happened right after the connect promise resolved OMEGALUL

#

The gateway closed with an unexpected code 1006

#

god i love the internet

dusty dove Apr 13, 2023, 1:24 PM

#

stable hatch ``` [251] Identifying shard id: 251 shard count: 256 intents: 0 ...

do u have more logs for this shard

#

this looks like it double-destroyed

stable hatch Apr 13, 2023, 1:25 PM

#

dusty dove do u have more logs for this shard

funny you should ask that

#

📎 logs.txt

#

have fun

#

256 shards

dusty dove Apr 13, 2023, 1:28 PM

#

[251] Connecting to wss://gateway.discord.gg?v=10&encoding=json
[251] Waiting for event hello for 10000ms
[251] Preparing first heartbeat of the connection with a jitter of 0.5238143187542947; waiting 21607ms
[251] Waiting for identify throttle
[251] First heartbeat sent, starting to beat every 41250ms
[251] The gateway closed with an unexpected code 1006, attempting to resume.
[251] Destroying shard
[251] Connection status during destroy
[251] Connecting to wss://gateway.discord.gg?v=10&encoding=json
[251] Waiting for event hello for 10000ms
[251] Preparing first heartbeat of the connection with a jitter of 0.34028765017804896; waiting 14036ms
[251] Waiting for identify throttle
[251] First heartbeat sent, starting to beat every 41250ms
[251] Identifying
[251] Waiting for event ready for 15000ms
[251] Ready
[251] Identifying
[251] Waiting for event ready for 15000ms
[251] More than one auth payload was sent.
[251] Destroying shard
[251] Connection status during destroy
[251] Connecting to wss://gateway.discord.gg?v=10&encoding=json
[251] Waiting for event hello for 10000ms
[251] Preparing first heartbeat of the connection with a jitter of 0.49264618846197816; waiting 20321ms
[251] Waiting for identify throttle
[251] Identifying
[251] Waiting for event ready for 15000ms
[251] Ready
[251] First heartbeat sent, starting to beat every 41250ms

#

got it

#

oh noo

#

i didnt extract everything if its multiline

#

ugh

stable hatch Apr 13, 2023, 1:29 PM

#

i think i can guess the issue

#

the identify throttle wait is never cancelled

#

even if the shard dies

dusty dove Apr 13, 2023, 1:29 PM

#

ahhhh

#

yeah

#

I just saw it too

#

oh no

#

kek

stable hatch Apr 13, 2023, 1:29 PM

#

PepeHands

#

good luck

#

sounds like HELL to handle

dusty dove Apr 13, 2023, 1:30 PM

#

yeah this looks like I need those abort controllers passed into the async queue

#

ThisIsFine

stable hatch Apr 13, 2023, 1:30 PM

#

~~which you can do but lord.~~

dusty dove Apr 13, 2023, 1:30 PM

#

actually no

#

i can just do a Promise.race in the shard

#

and it should be enough

#

worst case what ends up happening is the shard after waits a bit extra

#

if i dont do proper aborts

#

though

#

i could

#

nah ill just do my favorite "hack"

#

        this.debug(['Waiting for identify throttle']);

        const controller = new AbortController();

        const interrupted = await Promise.race<boolean>([
            this.strategy.waitForIdentify(this.id).then(() => false),
            once(this, WebSocketShardEvents.Closed, { signal: controller.signal }).then(() => true),
        ]);

        if (interrupted) {
            this.debug(['Was waiting for an identify, but the shard closed in the meantime']);
            return;
        }

        // clean up the once listener
        controller.abort();``` @stable hatch lol

#

should be fixed now

stable hatch Apr 13, 2023, 1:38 PM

#

LOL

dusty dove Apr 13, 2023, 7:34 PM

#

(we did this properly after all since kyra was moaning about it)

stable hatch Apr 13, 2023, 7:55 PM

#

~~why do people have to moan to do something~~

dusty dove Apr 14, 2023, 8:31 PM

#

merged

#

just need to wait for release now

dim oracle Apr 14, 2023, 9:04 PM

#

Alright, will probably test on Monday though

sullen snow Apr 15, 2023, 8:15 AM

#

https://safe.saya.moe/6fd72iliioxi.png thats a lot cleaner, lets hope this works KEKW

rare shard Apr 15, 2023, 8:26 AM

#

We also added an AbortSignal parameter, I hope you can handle it someway, @sullen snow

sullen snow Apr 15, 2023, 8:27 AM

#

yeah I can probably connect that in some sort

dusty dove Apr 15, 2023, 8:27 AM

#

lol i mean

#

if you don't handle it

#

things will break

sullen snow Apr 15, 2023, 8:27 AM

#

Scary means I need to reconfigure the thread to throw errors and handle the abort signal eh

dusty dove Apr 15, 2023, 8:27 AM

#

the reason why we needed it was because apparently if a shard closed while it was waiting for an identify

#

the shard would duplicate its connection

#

PepeLaugh

#

yeah just look at how I do it in the worker sharding strategy

#

you can probs follow a similar pattern

sullen snow Apr 15, 2023, 8:28 AM

#

though on your case its just passed on async queue?

#

then let the function throw an error

dusty dove Apr 15, 2023, 8:28 AM

#

yeah

#

but I meant how it gets there

#

since it's cross-thread

#

https://github.com/discordjs/discord.js/blob/main/packages/ws/src/strategies/context/WorkerContextFetchingStrategy.ts#L85-L100

#

https://github.com/discordjs/discord.js/blob/main/packages/ws/src/strategies/sharding/WorkerShardingStrategy.ts#L333-L345

#

though now that i think about it

#

you are still using the worker strategy

sullen snow Apr 15, 2023, 8:30 AM

#

with our scale only worker sharding is viable

dusty dove Apr 15, 2023, 8:30 AM

#

you just need to add the param to your throttler

#

and figure out how it should work

#

i guess

sullen snow Apr 15, 2023, 8:30 AM

#

my confusion just rises from, what does the abort signal does

#

when it emits, and how it should interact with waitForIdentify

dusty dove Apr 15, 2023, 8:30 AM

#

when controller.abort() is called the signal fires an event

#

in my case that's handled in the async queue and I just let it throw

sullen snow Apr 15, 2023, 8:31 AM

#

cause our identify handling never really needs to be cancelled, it would just clear up, then let other shards get the identify

dusty dove Apr 15, 2023, 8:31 AM

#

that's still a cancel though

#

like

sullen snow Apr 15, 2023, 8:31 AM

#

so my options is a, when abort signal is here, abort the thread waiting and reject the promise

dusty dove Apr 15, 2023, 8:31 AM

#

shard starts connecting and needs an identify

#

and then the shard tells you it no longer wants the identify

#

since the connection closed

#

that is cancellation

#

it frees up the identify it wanted for the next shard in line, though, yes

rare shard Apr 15, 2023, 8:32 AM

#

It's better to somehow handle the AbortSignal in some way because it lets you clean up resources and let the next entry go thru asap

#

If you have a blocking mechanism like a queue, and you don't free it up for a cancelled entry, you'll block the following entry unnecessarily

sullen snow Apr 15, 2023, 8:33 AM

#

and d.js manager also needs to know if the waitForIdentify throws an error?

dusty dove Apr 15, 2023, 8:33 AM

#

yes

rare shard Apr 15, 2023, 8:33 AM

#

It's true many APIs don't support AS, but a lot of them support a way to cancel/abort

dusty dove Apr 15, 2023, 8:34 AM

#

if you get told to abort

#

you need to throw

#

and free your lock so another shard can grab that identify

#

that's basically it

sullen snow Apr 15, 2023, 8:34 AM

#

hmmGe we'll see how I can cook it up on the redis sinec thats also offloaded on another thread

#

cause for some magic node.js reason

#

while loop while even the function inside of it is async

#

blocks the event loop

#

one last thing

#

signal: AbortSignal this is just the signal and not the whole abort controller class?

dusty dove Apr 15, 2023, 8:36 AM

#

yup

#

though the global types on it are incomplete for some reason

#

I had to hack this in

#

// Because the global types are incomplete for whatever reason
interface PolyFillAbortSignal {
    readonly aborted: boolean;
    addEventListener(type: 'abort', listener: () => void): void;
    removeEventListener(type: 'abort', listener: () => void): void;
}```

#

and I do (signal as unknown as PolyFillAbortSignal).addEventListener('abort', listener);

#

lol

sullen snow Apr 15, 2023, 8:37 AM

#

do this emit an event of some sort

dusty dove Apr 15, 2023, 8:37 AM

#

yes, abort

#

when controller.abort() is called signal's abort event fires

sullen snow Apr 15, 2023, 8:37 AM

#

oh, ok that makes things a bit easy I guess

#

KEKW

#

so in your code, its just passed on the "queue" instance, then when abort is called, the queue will reject, then throw the promise on where waitForIdentify is called?

dusty dove Apr 15, 2023, 8:38 AM

#

yeah, basically

sullen snow Apr 15, 2023, 8:39 AM

#

ok thanks I'll cook something up :

sullen snow Apr 15, 2023, 8:47 AM

#

dusty dove yeah, basically

https://safe.saya.moe/o0u3crqcu65u.png here I assume this is that, also one thing, if waitForIdentify throws an error, what happens?

#

also forget that promise reject, was due to original impl. without the abort signal KEKW

dusty dove Apr 15, 2023, 8:49 AM

#

sullen snow https://safe.saya.moe/o0u3crqcu65u.png here I assume this is that, also one thin...

https://github.com/discordjs/discord.js/blob/c87e8260878e1ec66cbc148927119670ff3ceb34/packages/ws/src/throttling/IIdentifyThrottler.ts#L8

sullen snow Apr 15, 2023, 8:49 AM

#

yes what I mean on that shard what it will do

#

reconnect, or leave it hanging

dusty dove Apr 15, 2023, 8:50 AM

#

well it should never throw in the first place unless the shard aborted it

#

lol

#

https://github.com/discordjs/discord.js/blob/c87e8260878e1ec66cbc148927119670ff3ceb34/packages/ws/src/ws/WebSocketShard.ts#L361-L375

#

and it only aborts it if it closed in the meantime

#

so it's already reconnecting by that point

sullen snow Apr 15, 2023, 8:51 AM

#

since its return, oh nvm, you have another handler on closed do you?

sullen snow Apr 15, 2023, 9:09 AM

#

https://safe.saya.moe/ar5u7cl35jir.png this should do it thanks

rare shard Apr 15, 2023, 9:38 AM

#

sullen snow https://safe.saya.moe/ar5u7cl35jir.png this should do it thanks

I'm curious, idk what promisify.send is, but does it not support a signal/is not trivial to do?

sullen snow Apr 15, 2023, 12:38 PM

#

rare shard I'm curious, idk what `promisify.send` is, but does it not support a signal/is n...

just a personal class for making promise based ipc, now you mentioned it, I could technically

sullen snow Apr 15, 2023, 2:22 PM

#

thanks for the idea actually, it looks a lot cleaner if the promisify class handles the cancellation internally

#

rare shard Apr 15, 2023, 2:51 PM

#

No problem ^^

stable hatch Apr 15, 2023, 7:16 PM

#

sullen snow https://safe.saya.moe/6fd72iliioxi.png thats a lot cleaner, lets hope this works...

You could really bump shardsPerWorker to like 4

#

Otherwise you're spawning 270? threads?

dim oracle Apr 15, 2023, 10:39 PM

#

Not sure how the math works but we've got 1440 shards with shardsPerWorker at 1

rare shard Apr 15, 2023, 11:16 PM

#

dim oracle Not sure how the math works but we've got 1440 shards with `shardsPerWorker` at ...

Then you have... 1440 threads

dim oracle Apr 15, 2023, 11:27 PM

#

rare shard Then you have... 1440 threads

That's seems a bit many? I don't know much about our websocket stuff but would that still be the case when we've got a cluster setup? (Basically like Kurasuta)

sullen snow Apr 15, 2023, 11:35 PM

#

i could but then again, i really like each websocket to have its own thread so i can be assured it is as fast at it can be

#

may change in future but the memory penalty is so ineligible

rare shard Apr 16, 2023, 5:24 AM

#

dim oracle That's seems a bit many? I don't know much about our websocket stuff but would t...

No, Kurasuta uses the amount of cores as many threads

#

Spawning 40 threads on a 10 core CPU won't make it magically perform better than spawning 20

dim oracle Apr 16, 2023, 5:47 AM

#

sullen snow i could but then again, i really like each websocket to have its own thread so i...

I mean if there's no performance increase if we go above the CPU's thread count then we might as well just use the same amount of threads as that the CPU has

rare shard Apr 16, 2023, 5:59 AM

#

There's so much the hardware can do

#

The reason why Kyoso can run 1440 threads and be just fine, is because /ws is very lightweight and uses very little resources per worker

#

Think about it, the reason you two run so many threads, is because sockets blocking other sockets on the same thread

#

But if you have a 10T CPU, it can only run 10 workers at a given time, if you have 1440 workers, 1430 will be idling and waiting for the OS's scheduler to give them a chance to run, and that happens way more frequently than if you have a worker per CPU thread (virtually almost never, all sockets would basically run with little to no stop)

stable hatch Apr 16, 2023, 6:19 AM

#

I mean that said, cramming 1440 in like 10 threads is also not ideal

#

Speaking from experience, cramming that many ws connections (~144/thread) in one single process/thread will cripple it

rare shard Apr 16, 2023, 6:22 AM

#

Realistically, such large bots are likelier to run on servers with a lot more cores. If they have 64 cores, 1440 will require only 23 (rounded up 22.5) sockets per thread

#

And many services force you to increase CPU core count when increasing RAM count, so to account for the RAM needs large bots need, together with the amount of CPU required to run so many sockets...

stable hatch Apr 16, 2023, 6:23 AM

#

64 cores is not cheap

#

Anywhere

#

My bigger worry is if threads stay alive when the parent dies

rare shard Apr 16, 2023, 6:24 AM

#

The only positive side I see of running more workers than CPU threads, is that the GC has less memory to sweep

stable hatch Apr 16, 2023, 6:25 AM

#

Not like shard threads keep much in ram that needs GC

rare shard Apr 16, 2023, 6:25 AM

#

But even so, depending on how the objects are managed, it's possible and likely that the scavenger deals with basically almost all the objects

#

So the GC does little to nothing

dim oracle Apr 16, 2023, 6:38 AM

#

We have 36c/72t

#

But CPU isn't really an issue

stable hatch Apr 16, 2023, 8:53 AM

#

Nice sin wave

dim oracle Apr 16, 2023, 9:06 AM

#

thanks

stable hatch Apr 16, 2023, 10:19 AM

#

dim oracle We have 36c/72t

So you can have ~20 shards per worker

#

Wouldn't recommend it

#

I'd go for max concurrency then

dim oracle Apr 16, 2023, 10:19 AM

#

I mean we used to run all just fine

stable hatch Apr 16, 2023, 10:20 AM

#

That's 1440 shards in one thread

#

Badddd idea

dim oracle Apr 16, 2023, 10:20 AM

#

nah we have clusters, so probably 24 per worker?

stable hatch Apr 16, 2023, 10:20 AM

#

I'd go for powers of 2

#

..aka 16

#

Or 32

#

Whichever floats your boat

#

Tho you have uh, around 90 shards per ratelimit key

dim oracle Apr 16, 2023, 10:23 AM

#

eh I'm not concerned about the startup/concurrency ratelimits

#

that'll work fine in whatever shardcount/shards per worker I run

stable hatch Apr 16, 2023, 10:25 AM

#

Fair

rare shard Apr 16, 2023, 10:27 AM

#

I think 16 is a sweet spot

#

But then again WS is capable of handling far more than one would give credit for, my main bot runs 14 internal shards in Discord.js v13, and Discord.js puts a heavy overhead on every message it gets, plus it's using a lot of intents, yet it's handling everything fine and the event loop latency is unnoticeable

#

Raw /ws should be able to handle much more, so I think you'd be fine even with 20 shards per worker

dim oracle Apr 16, 2023, 10:30 AM

#

this is our eventloop with 1 shard per worker atm

dusty dove Apr 16, 2023, 10:30 AM

#

me when still no custom rust erlpack

rare shard Apr 16, 2023, 10:31 AM

#

Not like erlpack would fix the performance, DD meguFace

#

The optional WS dependencies do, tho

dusty dove Apr 16, 2023, 10:31 AM

#

no i know im just harassing you for not doing things

rare shard Apr 16, 2023, 10:31 AM

#

~~undici.ws would too, hopefully~~

dusty dove Apr 16, 2023, 10:31 AM

#

thats not stable yet is it

#

last i looked at it it def wasnt useable lmao

rare shard Apr 16, 2023, 10:31 AM

#

I don't think it's released

#

Oh I think that even if it's released, the performance would suck because it uses the McBloaty web event API, which has even worse performance than pre-fixed AEE

sullen snow Apr 16, 2023, 12:00 PM

#

dim oracle this is our eventloop with 1 shard per worker atm

it should not affect anything

#

even you run 1 or all

#

but then since we now use json encoding, cpu usage is definitely a thing now

#

I did have experience on original d.js ws also your original code before I refactored it where the bot is literally screaming on cpu usage due to the js event loop getting overloaded by json.stringify and json.parse, and erlpack did fix it. we just dropped it now because you can run the ws threads in its own "thread"

#

besides I don't see any issue on running a bit of threads on a dedicated container since most of our pcs anyways even handle more than that amount of threads without issues

#

like even with 56 shards / cluster, discord.js was able to run flawlessly with 1 shard per thread on /ws

#

reference: https://grafana.saya.moe/d/kashima-is-a-good-girl/kashima-cluster?orgId=1&refresh=5s https://safe.saya.moe/zcltakyivkcw.png

would drop this down to 4 clusters again, but then again the memory penalty is very unnoticeable to the point I'd rather just have each ws on its own event loop to ensure their stability

dusty dove Apr 16, 2023, 12:22 PM

#

yeah ure lucky im so hot and talented :^)

sullen snow Apr 16, 2023, 12:23 PM

#

dusty dove yeah ure lucky im so hot and talented :^)

will be saving that until the optional deps (bufferutil and utf8 validate) is on /ws KEKW

dusty dove Apr 16, 2023, 12:23 PM

#

?

#

they work

#

i dont need to do anything special to "support" those

sullen snow Apr 16, 2023, 12:24 PM

#

i dont no / think so (?)

dusty dove Apr 16, 2023, 12:24 PM

#

it's purely on the ws package

#

if you install them ws uses them

#

it just wasn't in the README until some point

sullen snow Apr 16, 2023, 12:24 PM

#

not sure since idk if ws even use it

#

KEKW

#

but anyways, on that kind of guild / process, stock d.js will also not work

#

even with probably less cache, still it may not work

dusty dove Apr 16, 2023, 12:24 PM

#

https://sucks-to-b.eu/wLJEYD.png

sullen snow Apr 16, 2023, 12:24 PM

#

the amount of overhead d.js have will just make the whole process blow

#

can you even check if the ws use that

#

not the /ws package but the ws package itself

dusty dove Apr 16, 2023, 12:25 PM

#

.. it does

#

this is literally how discord.js "implements" bufferutil and utf-8-validate too

#

we don't do anything

#

we just tell you you can install them and they'll be used

sullen snow Apr 16, 2023, 12:26 PM

#

oh well, probably it works then

dusty dove Apr 16, 2023, 12:26 PM

#

https://sucks-to-b.eu/bbyAun.png

#

https://github.com/websockets/ws#opt-in-for-performance

sullen snow Apr 16, 2023, 12:26 PM

#

can't tell but yeah

#

i keep both the buffer util and utf 8 on my package file

#

so I guess ws should use it

stable hatch Apr 16, 2023, 3:33 PM

#

sullen snow I did have experience on original d.js ws also your original code before I refac...

Your issue was not using zlib-sync

#

Json parse can be very fast

stable hatch Apr 16, 2023, 3:52 PM

#

sullen snow so I guess ws should use it

Yep, you can see it in ws/lib/buffer-util.and ws/lib/validators

sullen snow Apr 17, 2023, 2:47 AM

#

stable hatch Your issue was not using zlib-sync

but then again is the gc spikes fixed on that

#

its crucial for us to be stable, and until I got a verification on that gc spikes now fixed on that, we can't run it in prod

stable hatch Apr 17, 2023, 5:33 AM

#

Aren't you guys already running it

stable hatch Apr 17, 2023, 6:48 AM

#

Also you wouldn't notice the gc spikes (if there's any) since they're on different threads

sullen snow Apr 17, 2023, 10:59 AM

#

stable hatch Also you wouldn't notice the gc spikes (if there's any) since they're on differe...

yes but since it increases over time, it will eventually lead to a very slow thread

stable hatch Apr 17, 2023, 10:59 AM

#

Hmm, strange

sullen snow Apr 17, 2023, 11:00 AM

#

i have an old message here

#

showing the issue

#

#archive-offtopic message

#

what we used is erlpack which works fine

rare shard Apr 17, 2023, 11:02 AM

#

Oh, so zlib-sync makes the GC go brr?

sullen snow Apr 17, 2023, 11:02 AM

#

yes if nothing is fundamentally changed when I noticed that issue

rare shard Apr 17, 2023, 11:03 AM

#

I'll probably look into making a replacement for that library using CF's zlib somewhere this summer, same for erlpack (but in Rust), although the latter will probably happen first

stable hatch Apr 17, 2023, 12:44 PM

#

Etf shouldn't be the priority @rare shard

rare shard Apr 17, 2023, 12:45 PM

#

I know it isn't, but it's more fun to write

stable hatch Apr 17, 2023, 12:45 PM

#

https://github.com/discordjs/discord.js/issues/9075#issuecomment-1477462108

upbeat ermine May 8, 2023, 2:35 AM

#

dusty dove ``` [251] Connecting to wss://gateway.discord.gg?v=10&encoding=json [251] Waitin...

What was the solution to this? I also got this error code 1006 issue 👀

#

If I am looking at my logs right, this error seems to cause the rest of my shards to eventually spiral out causing a full bot crash, but I could be misinterpreting this as the main cause.

dusty dove May 8, 2023, 6:31 AM

#

upbeat ermine What was the solution to this? I also got this error code 1006 issue 👀

please dont necro a month old issue

#

if you're having issues id prefer you open them on github with full logs

#

i cant keep track of so many people in the discord

dim oracle May 8, 2023, 7:18 AM

#

While this thread has been reopened I might as well mention that we haven't had any issues since the last one 🙏

#

It also seems like a few more people joined in

dusty dove May 8, 2023, 7:26 AM

#

i dont mind

#

its just like, i want to limit this thread to specific convos

#

but now that /ws is in mainlib

#

i dont want like everyone to come with issues here

#

github is still best when theres many of them so i can track it

upbeat ermine May 8, 2023, 7:31 AM

#

I apologize, I was just searching for similar errors and came across this. I figured since it was relating to big bots, and with my bot also being a big bot, this would be an appropriate place to ask. I've made some comments on this Github issue:
https://github.com/discordjs/discord.js/issues/9139

GitHub

Shard Crashes no Error · Issue #9139 · discordjs/discord.js

Which package is this bug report for? discord.js Issue description Running startup on either PM2 or general shard manager leaves the same problem, on a docker container if that's any use to kno...

dim oracle May 8, 2023, 7:53 AM

#

upbeat ermine I apologize, I was just searching for similar errors and came across this. I fig...

The logs you've posted show a completely normal WS shard resume so unless there's logs you haven't shown then I don't think that's the right issue, if you really think it is related to d.js then its probably best to create a new issue with more info.

Also anything above 150k guilds is considered a "big bot" since you will get access to big bot sharding then ^^

dusty dove May 8, 2023, 7:54 AM

#

all of the above

dim oracle May 8, 2023, 8:05 AM

#

side note, I will start to donate a bit now and then via Open Collective and I have stopped supporting on Hydra's Patreon so I'll probably lose the sponsor role, anyhow first $200 is sent for now

dusty dove May 8, 2023, 8:10 AM

#

dim oracle side note, I will start to donate a bit now and then via Open Collective and I h...

we grant the sponsor role over OC contributions anyway

#

if its auto removed by patreon just DM crawl w a transaction from OC and he'll grant it back

#

that's the process

#

oh actually we have a command for it now

#

https://sucks-to-b.eu/1yvlfP.png

dim oracle May 8, 2023, 8:11 AM

#

The Patreon bot is pretty ass anyways so I doubt it'll actually remove it correctly

dusty dove May 8, 2023, 8:11 AM

#

yeah lol

dim oracle May 12, 2023, 10:37 AM

#

Alright so we seem to have another issue, a small spike where a lot of shards disconnected and resumed but in all of this one shard never recovered and is stuck reconnecting since then. These are the logs around that specific shards, the ECONNRESET is most likely related but not 100% sure. After these logs the shard is never seen again and is stuck reconnecting like I said.

📎 logs.txt

#

There were about 11 ECONNRESET errors around that time

dusty dove May 12, 2023, 11:01 AM

#

i love networking issues

dusty dove May 12, 2023, 11:03 AM

#

dim oracle Alright so we seem to have another issue, a small spike where a lot of shards di...

wait so its never seen again in logs as in

#

"waiting for ready" is the last thing you got?

dim oracle May 12, 2023, 11:05 AM

#

Yeah

#

I don't really have a decent way to access all of the logs (log file is 60gb atm) but the "waiting for ready" line is at 06:13:17 and the snippet I got goes until 08:30:55

#

and no mention of that shard within that timespan

dusty dove May 12, 2023, 1:03 PM

#

gotcha

#

yeah ive seen some similar behavior described

#

im a bit unsure whats going on there

#

realistically that should timeout and throw

#

all signs point to this just being the worker dying entirely

#

but im unsure why

#

if it dies to an unhandled exception unless you guys messed with the worker strategy that should cause your main process to re-throw the error and therefore quit entirely

#

but i cant imagine what it could be dying to that isnt an unhandled exception

#

sounds like a gracious exit?

#

@dim oracle do you guys have your own worker strategy still

#

think you could do me a favor and attach an exit log to the worker since i dont

#

and log when it fires

#

and when you run into this again check if it did fire

dim oracle May 12, 2023, 1:20 PM

#

cc @sullen snow

upbeat ermine May 12, 2023, 11:46 PM

#

dim oracle Alright so we seem to have another issue, a small spike where a lot of shards di...

I have this exact same issue with my bot
So it's definitely not just you if that is helpful :)

dim oracle May 13, 2023, 6:08 AM

#

upbeat ermine I have this exact same issue with my bot So it's definitely not just you if that...

Would be cool if you could include some more info you might have gathered from debug logs or anything ^^

#discordjs/ws big bot memes (old)