#I noticed my app appeared to miss some events

44 messages · Page 1 of 1 (latest)

iron pumice
#

100% this could be a bug on my side but I noticed I stopped getting events, based on my logs it looks like this started sometime around 2023-07-26T22:30Z, I don't have an exact time. I did just try tweak the targets and that message did not show up. Restarting my app appears to have fix the problem. Since I'm not seeing other repots I guess it is a bug on my end. I just wanted to post to see if others saw anything similar.

cinder yacht
#

Did you have any disconnect event?

#

@iron pumice

iron pumice
#

Nope

#

Based on my logs it looks like I just stopped getting pings (and all other events).

#

Ooo I found found the time of the last ping, so it looks like my original estimate of late on the 26th was off by about 2.5 hours 😞. the 18:30 on the 27th final log entry from the app running about 10 days straight., I'm fairly sure that was the longest continuous run for my app, so there could be a bug on my side that only shows up after running for 9+ days 🙃. I'm not currently tracking when Pongs are getting sent back. I'll likely add that.
[2023-07-27T18:33:40.6308690Z-10.00:00:02.8643540] Last Ping: 2023-07-27T00:09:47.4897646Z

cinder yacht
#

When you don't get ping, you should have an automated disconnect message with a reason from SocketIO, so that's strange

#

or a connection_error

#

One of both

tough basin
#

hmm. Ive also been seeing this issue.

#

I have a bunch of error handling on the container. And the websocket client never actually raises an error. It just stops and sits there doing nothing.
I was also going to add logging for the polls

#

Keep in mind my container is very simple. all it does is listen to all events and push everything to a webhook, liklihood of it being something with my code is very low. Since there is very little code to begin with.

#

what client language are you on?

cinder yacht
#

I have a bunch of error handling on the container. And the websocket client never actually raises an error. It just stops and sits there doing nothing.
I could see if I get notification on our side of your disconnection Think

#

Did you both add disconnect, error, connect_error handlers?
Maybe for some reason you're disconnected and you can't connect again Think

iron pumice
#

There was a disconnect within that 10 days of run time, but the app detected that one it took a while (like more than a hour) but it was able to reconnect. And things did return to normal it got some evens, and then radio silence. I may try adding some more detection/recovery (aka after not getting a ping for a while I may try to reconnect automatically)

cinder yacht
#

It disconnect if you don't get the pong, but then it should try to reconnect, as it's not a connection error. The only handler that you get where it's not auto reconnect is connection_error. Or when you out of reconnection attempt

#

If it happened again, ping me, and I will check on my side if I see anything Pray

tough basin
# 
[email protected]:26257/ws2wh> select event, created_on from websocket_log order by created_on desc limit 20;
  event  |         created_on
---------+-----------------------------
  attack | 2023-08-02 00:24:26.489345
  attack | 2023-08-02 00:23:11.363682
  attack | 2023-08-02 00:22:19.62381
  attack | 2023-08-02 00:21:23.660681

Stopped receiving events at 2023-08-02 00:24:26.489345 UTC
Even though the program is still running

# 
[email protected]:26257/ws2wh> select event, created_on from websocket_log where event = 'connected' order by created_on desc limit 1;
    event   |         created_on
------------+-----------------------------
  connected | 2023-07-28 21:27:46.607481

Can also see the last time I was forced to restart it was on the 28th. so it lasted about 4 days without issue

#

It also turns out I don't have disconnect/error/connect_error handlers because those are not included in the catch-all event. So I'll need to do that. But from reading the docs, these are just for logging, the reconnect logic is handled internally within the lib

cinder yacht
#

I will check check

#

Send me the app token you're using @tough basin in DM

cinder yacht
#

I don't see any error on our side. Do you still get the ping pong event?

#

Next time you see the issue, if possible, don't restart your bot and ping me, so I can look deeper, now I can't really check anything more because you're not in this strange state Pray

tough basin
#

hmm ok. that's helpful to me since it shows it was still connected. But not receiving events.
It's annoying to not reset since that's data I'm missing out on, but is what it is I guess.

#

I don't track the acks. I need to fork the lib to do that

cinder yacht
#

I do have some idea of what could eventually happened, but yeah, only way to verify it is live (I mean, the consequence, not the cause, but I need to know the exact consequences first to identify what cause it)

tough basin
#

@cinder yacht

[email protected]:26257/ws2wh> select event, created_on from websocket_log order by created_on desc limit 10;
  event  |         created_on
---------+-----------------------------
  attack | 2023-08-02 18:33:00.433529
  attack | 2023-08-02 18:30:43.869574
  attack | 2023-08-02 18:29:52.346956
  attack | 2023-08-02 18:29:05.285724
  attack | 2023-08-02 18:27:56.214769
  attack | 2023-08-02 18:27:35.160889
  attack | 2023-08-02 18:26:50.192556
  attack | 2023-08-02 18:26:39.405808
  attack | 2023-08-02 18:25:54.978604
  attack | 2023-08-02 18:25:34.49403
(10 rows)

Appears to have stopped

#

same token

cinder yacht
#

I'm checking in one hour

#

@tough basin what package are you using for the client ?

tough basin
cinder yacht
#

@iron pumice Are you also using the python client?

#

@tough basin For now I still see you conneted on our side.
What would be useful to verify is on your side:

  • Do you still get the ping/pong event
  • Enabling the logger and the engineio_logger
  • Check if you don't catch any unexpected error in your code that can be due to sub process (database for example)
  • Print what your save on the database, so you can check if you get still event, but it fail elsewhere in the process

Actually on our side, it look all fine. There will be a few tools that comming in the next weeks that will help me to check further, but immediataly, I don't see any issue. And if it was on our side, we would have way more report (me included through Darkbot).
So I opt in for an issue with the python client, or an unexpected bug on your side for now.

Did you do some recent update in your code? That's strange that start sudently

#

Send me the clan code you're following too Silical please, I will have another look a bit deeper

tough basin
#

I think you are actually right

#

i'll need to check if this client exposes a route for handler errors

cinder yacht
#

There is one on the server side, which is on_error_default decorator. Might check if there's one on the client. Let me know what you found out, might be useful to know

cinder yacht
iron pumice
cinder yacht