Desync bug

[peak nacelle]

https://www.youtube.com/watch?v=8IICIeXyTIM&ab_channel=JAWSMUNCH304

https://www.youtube.com/watch?v=8IICIeXyTIM&ab_channel=JAWSMUNCH304

Compare the replays (my pov in the replays above) to jaws twitch vod. Different things happened in all 3 games in the twitch vods and in my games, where only I would see the input commands from the enemy, but they couldn't see my input commands at all (Almost as if I'm playing as an afk guy from a replay.) It would trigger randomly around 5-10 minutes in each game.

Weirdest one was the Glitters game where around minute 9 or 10, I stop preceaving input commands from all other players, but on my screen Im killing their bases, while on his VOD, my innput commands arent being transmitted

This happened all 3 games

[naive marsh]

We had an issues where players would see different things occurring in game after a "Sync error" notice appeared. From one player's perspective other players would go afk, although his game commands and chat would be still working. For the other players, he would go afk. In my case, the replays would have the same issuies, being seeing from the desynced player's perspective. Weirdly, the ingame chat would still be working normally, which created more confusion. Refer to Jawsmuch304's youtube video (from 1h40min to 1h59min) vs the first replay posted in by Kellar (Glitters v2).

[void pulsar]

yes just experienced that for 3 games in a row

my forces just advanced and killed enemy base.. but was desynced..

when i reloaded.. it still looked like that..

i had to restart BAR

to load the real game where they actually sat there outside the t2 fac and did nothing

but my ally took and finished the game

[naive marsh]

At this point this is happening in half the game I play or spectacted today

[north kindle]

I also experienced this bug multiple times today

[elfin birch]

I am here to report the same bug. Half of each team was desynched from the other half. In game it appeared that I was in the desynched group, so I checked my replay and it was what I saw in game. So I downloaded the replay from the website and it's pretty much identical, with some differences in chat.
But in the downloaded replay, the desynch messages were coming from the other side.

Players were getting kick-banned over it when they did nothing wrong.

game:
Beyond All Reason test-26834-34e0581

[dim birch]

+1

[north kindle]

It happend again, it shows a message on top "xxx player desynced frame xxx"

Is it the client version vs the server version the issue?

[rustic viper]

i saw this bug showed up today in multiple matches to different players.

[cursive glen]

yes, 5 or so matches that i have played and watched had the desync issue; its like every one is playing on their own and from client side, every one is idle, except the client player.

[crystal zephyr]

Well same bug happened to me

And now I am delusional

[peak nacelle]

Watching the replays now, they are still showing it from their perspective, which is weird.

[keen dawn]

I started desyncing ever since I logged into website o.O

random dog asked how to remove his flag from the game - so I logged in just for that reason and basically 3 games afterwards desync

oh and that dog desynced as well first tgame if I remember right

[brittle gust]

I've also had this issue for the first time. Man I was thinking I was playing like a 60 os wiped the entire team myself, but uhh... I've been informed that was not the case lol.

[elfin birch]

Third time today, seen multiple people getting kickbanned, and matches 20 minutes in and lost before anyone knows what's happening. Maybe you guys should consider rolling back OS scores. Even better if you can just negate any games with desynchs in the last day and a half. It's causing a lot of animosity. I've seen N words and F slurs being thrown around over this.

[candid berry]

We reverted a few things. Let us know if the desyncs are still happening after lobby refresh

[keen dawn]

still desyncs

[dull drum]

I still get someone desyncing in almost every game

[peak nacelle]

Havent had any issues that ive seen past 3 hours (3 games)

[supple sphinx]

seen two players get desynced midgame

[worn fern]

I had that issue today on Glitters, weird thing is I was playing FFA battle for 130 mins before and no desync occured

[candid berry]

What we are looking for is an infolog from a dude who didn't have a desync occur while everyone else in the lobby did.

@proven inlet right?

[proven inlet]

When a desync occurs, game states get generated for all clients. It is in the same folder as the current infolog. We need the game state of a good and bad case for the same match.

[minor wasp]

upload log in launcher will also grab the game state, so you can use that to upload quickly.

[strong thorn]

fwiw if it helps fixes things, I get desyncs consistently from some lobbies and not from others, my guess is that there's some non-backwards compatible change that got pushed and is causing desyncs btwn lobbies with players on old clients and new ones

[minor wasp]

It's not possible to start game if any 2 players don't have exactly the same version of game, map, engine

[stray vessel]

I just had a lobby where half the players de-synced too, is there something we can do about this? Like upload logs or any more info about this issue/bug?

It actually ended the game for my while others kept playing on as at some stage it seemed like I just randomly desynced too

[candid berry]

This? Apparently. It's a bit too big brained for me to understand.

[peak nacelle]

Just happened again in Arrakis Light lobby

GG

[willow totem]

@chilly swan you were in this game which I desync'ed in; can you grab the ServerGameState--XX.txt and ClientGameState--.txt from your *\BAR\Data\* folder and drop them here, as Tarnished said above these are needed for troubleshooting from both a sync'ed and desynced player

If you open the file and check these match gameID: ba280c67206948fd681fa77b2da76013

This match was !stop'ed after only 2 minutes so its very short with not a lot going on

[crystal moat]

https://log.beyondallreason.dev/logs-20241013T2136-BNvU4dlV.zip

[dim birch]

https://log.beyondallreason.dev/logs-20241013T2136-iparwJGd.zip

These two are where one is desynced while the other is not

replay from my pov

[crystal moat]

same game - for me no desync

[elfin birch]

I'm suspecting there may be an issue with the launcher. I've been having big lag-times and crashes on the launcher start. I just joined a full lobby trying to start the match, and it wouldn't allow them to start because I was 'not synched'. I wasn't in the game, and it didn't even show me in the spectator list or queue.

I had just played in that same lobby, then after the match left and did some other stuff, came in to join after they'd played another match and that's when the above issue occurred

[minor wasp]

desyncs have nothing to do with launcher or lobby

descyns are purely game and engine related

[rustic viper]

This game is now a broken mess right now, there is not a single match without desync, i just saw a lobby where someone complained that 5 people in a match had de-sync issues. Why isn't there a engine rollback happening?

[glad ember]

consider rolling back to an older engine until this is resolved? For me personally, using a VPN to get an alternative network routing path solved my desync issues. This might offer some clue on how to resolve this.

[keen dawn]

just full rollback this is unsustainable

every game desync

it feels worse and worse

6 fuckin people desync in one game; that's new record or?

nvm 8 people got desynced; they literally stayed in their own game after game ended

xd

new record fr fr

[keen dawn]

@minor wasp little to no desyncs on glitters and strait; however, rotato is full of it...

The major difference is that roto players are addicted to rezing - glitty/ strait is reclaim only

Lava strait has desyncs as well (they rez there as well)

[brittle parrot]

Just had a game where several people were desynced, myself being one of them, we had to stop game after 10 minutes

never had that issue before

[willow totem]

Plenty on isthmus over the weekend

[forest holly]

Hi guys, apologies if I'm just adding to the noise here, but my friends and I played ~ 4 games on Isthmus EU last night UK time (about 12 hours ago) and each game had at least one desync, often multiple, the game was unplayable. Happy to help anyway I can, retrieve logs etc

[keen dawn]

Resurrection seem not to trigger desync.

[minor wasp]

@keen dawn @rustic viper: I was just communicating that it's neither lobby or launcher that can affect this so those should not be brought up. I'm not the person responsible for this problem, it's 100% outside of my area, I'm personally not going to do anything about it. There are other developers working on it.

[brittle parrot]

And I should note that for the first game, where multiple people desynced, I actually did an alt-f4 to close the game and then went back in and rejoined the game, caught back up, and was still desynced in the same fashion. So it's not a simple dropped packet or something, but the server repeatedly failed to give the proper data or my game repeatedly failed to process specific data, as it seemed to fail the same way both times. We had to stop the game. Then the next game had issues too, so we stopped that one as well and lobby died. Also it is weird that the message for a game breaking desync is so subtle I didn't even notice any such message when I was playing and it wasn't until I was asking for my t2 con and people were telling me it was in my base and my mexes were already upgraded that I realized we were no longer seeing the same thing and game was broken. I then joined a coop vs Barb AI game and I think a desync may have occurred in that game too as there was an unkillable mex that my units kept shooting to no effect (probably because it was long dead, but stuck alive in my game). Kind of weird that a desync would make a unit stuck alive on someone's game. Mentioning that specifically as it may aid in figuring out the nature of the problem.

Also of note the T2 con I think that was given to me just showed as sitting in the ally's base, still owned by him, on my end, but on everyone else's it was given to me and building stuff in my base that I couldn't see.

[reef kernel]

Please, revert the changes which causes this problems - it's better to lose the few new features and have a working game then to stay on it

[willow totem]

#⭐｜main message

[reef kernel]

NICE!

Really curious from a developer perspective what is/was causing this bug. Please keep us updated

[rustic viper]

I saw regularly desync on glitters too

[elfin birch]

After a desynch match where rejoining didn't fix it on my end, the downloaded replay from the website agrees with my perspective that I steamrolled the enemy team. People on both teams talked about it and reacted to it. But on the other side, some of them saw themselves steamroll us.

So even though the official replay shows my side where I won, the official results ON THE SAME REPLAY show their side win.
Beyond All Reason test-26848-3dfdfa4

Very weird that the official replay and the results listed on the replay disagree with each other.

[violet heron]

the plot thickens

[orchid plover]

And its kinda funny & mess up

[worn fern]

From your experience guys, do you have desyncs only on glitters?

[elfin birch]

no, also on Isthmus for sure. Someone else said it mostly happens on rotation lobbies, so I think it's everything

[silver light]

Happens on many maps.

[worn fern]

For me it has occurred only for glitters ... it is my luck 😄

[hoary whale]

@keen dawn There's a theory that enabling Legion fixes desyncs. Can you test?

#⭐｜main message

[tulip silo]

I have posted 2 different replay with desync yesterday, a small team game

[supple sphinx]

no one has said this here i think but even spectators get the desync bug

[hoary whale]

Does it happen when you go to player view of desynced player as spec?

[supple sphinx]

uh i didnt experience it but i did see red notif in the middle top indicate its one of the spectators

[willow totem]

I've desync'ed over the weekend while my client was specing afk looking at the whole map i'm pretty sure can't find the evidence of this happening that I thought I had - so take it as anecdote

but haven't desynced today at all

[tulip silo]

Do we have a suspected?(Commit)

[elfin birch]

If I quit and rejoin, it just replays the desynched version and doesn't fix anything.
But if you close the client and rejoin, you get back in the right version of the game.

[astral phoenix]

I'm a little confused trying to understand a replay im watching with desync errors. It says

[t=00:03:38.176060][f=0030610] Error: [DESYNC WARNING] checksum 94b8389c from demo player 0 (argo4242) does not match our checksum d3e7467b for frame-number 30600
[t=00:03:40.232827][f=0030671] Error: [DESYNC WARNING] checksum 25f9b683 from demo player 0 (argo4242) does not match our checksum 43cdb0c9 for frame-number 30660```

etc. etc.
However, the player is still with us... and someone else quit normal quit

and if I remember correctly I thought they were on a massive delay before leaving

https://www.beyondallreason.info/replays?gameId=70f20b6759e9319355d05c5146470ba8 This is the replay, Zaelon is who I thought desynced and timedout but now I'm not sure since errors are for argo4242 😄

[tulip silo]

last unranked 2v2 on ccr

[tulip silo]

only certain players

i watched tonight 5-6 matches and for example "fbc" desync every match

games was unrank so if you need the replay i have to manually upload

[sonic spoke]

See transport changes were never meant to happen 😇

[junior ridge]

Another game, another desync. We just had a desync were a small groups of players were in the same "desync bubble" AKA seeing the same game but the rest of the lobby was playing a completely different match.

[sonic spoke]

Expect this to go on until devs state changes made for temp resolve

Been happening for pst 48 hours at least

[junior ridge]

2 of us were in the same party and were seeing the same game. Rest of the lobby was in the non-desynced one. Really weird that we would both desync on the same thing and have the exact same sim though.

[willow totem]

Would more pairs of ClientGameState be helpful? We were both spec's in this game, 2 players had already desync'ed and then Syphos while speccing desynced also

well dang... I assumed there would be something different

[astral phoenix]

I think they want it from the player that desynced and a player that did not ref: #1294721680379351150 message

[sullen obsidian]

i once gave EVERYTHING i have to someone (cntrl A, share units)... only shared like 16
... so some stuff pass

[keen dawn]

@proven inlet have you found the reason?

@hoary whale I know how to reproduce

desync

@candid berry

Go camera settings -> ui.settings.option.select_spring
and set graphics at ultra
and zoom out/ zoom in
shift-select units and use command (right click random location)
-> desync

I just reproduced desync via this 3 times

hmm cant do anymore

after game reload

whenever desync happened it was upon zooming out - the game would freeze for a bit (something people were talking about)

into desync

but after reload zooming triggers no "lag/ freeze"

[keen dawn]

5 20 - Apex desync -> 2024 - 10-15 632 replay - when zooming
7 36 - FFA - SilverSan (in pTaq Raikoh/ Titanic) game - desync when zooming (super noob so quite ez to see)

--checking all replays here

Zoom out/ in is trigger for all my cases for now

[keen dawn]

Which goes hand in hand with people complaining in games that zooming out seems laggy lately

[proven inlet]

Replays contain the checksums reported by the clients. When you watch the replay, your machine generates checksums for each frame and compares them with what was originally reported by the clients. Those messages tell you that on this particular watching of the replay your machine disagrees with with argo4242's checksums.

[proven inlet]

The freeze is when the desync has been detected by the server and has requested that all clients deump their gamestate. The freeze is the time taken to dump the game state.

[keen dawn]

hmm but if freeze happens when zooming out on repeat then that should be a connection, no?

I mean I will check all replays - would be best if I got only noobs with little to 0 apm to see

[proven inlet]

@amber wren since you're still getting desyncs it may be best to redeploy a version of BAR from say last Thursday. Sometimes going backwards through older builds is the only way forward.

[proven inlet]

It may be the cause it may be something else happening at the same time, but it is incredibly rare for a UI element to cause a desync. Because only simulation can cause a desync. If the UI causes a desync then it did something to change the simulation state.

[keen dawn]

Zooming out is not it - kellar did not zoom yet got desynced

[hoary whale]

Is rolling back to an earlier version in git really that hard to do for this? Is it because we have too many dependencies that could break if we do it?

[proven inlet]

I'm hoping that the game team have a way of DEPLOYING without having to destroy their commit history!

If not, then they better start adding the ability to deploy from release branches.

Though aren't all recent builds kept in github?

You can point the launcher to the older build.

That's how we do it for the engine.

[keen dawn]

How the absolute fuck am I able to reproduce the desync 3 times in a row then restart client and nothing; heh

any way to move frames/ ticks in replay by 1 - instead of pausing like a dog?

[proven inlet]

Welcome to the wonderful world of desync hunting. 🙂 Desyncs as you can see are perhaps the worst kind of bug that can happen to that RTS game.

[keen dawn]

Kellar on Tetrad_V2 got desynced right at the moment of destroying LLT

exact moment

[proven inlet]

/skip f <frame num> is the way I know, but going through per frame like that is going to cause RSI !

[keen dawn]

The other dog "Sletox" got desynced right after zooming out enough to that LLT

[proven inlet]

Does the desync occur based on whether you have a particular unit selected or not perhaps?

By particular I mean, you have a unit selected/or not when it dies.

[keen dawn]

I will go more deeply for this bcs I dont understand how every desync happens on zoom, but kellar has no zoom happening

[keen dawn]

I will check more deeply

[proven inlet]

We've had a desync based on unit-selection in the past; though, that was fixed.

Actually that was because a unit was being followed by the camera....

[keen dawn]

Also something noteworthy - this llt got killed at the very edge of vision

[keen dawn]

@proven inlet uhm is it perhaps something outside of each individual game? There is one additional thing to note - once desync happens - it will happen in any other game you play/ spectate until you restart the game;

Reloading game will get you desynced;
Starting another game will get you desynced;

You have to restart the game to fix the issue... that's weird, no?

Perhaps something sticking in the memory/ thread... but that should get culled off after game ending...

Floris also basically desynced in all games he joined - once you experience desync - it gets sticky until you restart client... that's super sus

--> Yup - you have to restart whole client in order for desync to stop happening (like close whole bar) --> that's weird.

[amber wren]

there was one PR merged into chobby saturday but i dont think thats it either

https://github.com/beyond-all-reason/BYAR-Chobby/commits/master/

[keen dawn]

Wasn't there a provider maintenance before the desync started happening?

ovh had maintance... so perhaps.. but not sure when exactly

[proven inlet]

Can you republish a BAR version from Thursday and see whether the desync rate clears up? Then you can slowly roll forward until you find the offending commit.

[keen dawn]

@proven inlet shouldn't desync be connected to each individual game? Isn't it a weird that it is sticking till you restart the client?

[proven inlet]

Desyncs can be sourced by anything if it can somehow influence the simulation state.

Overclocking your CPU/memory can cause desyncs for example.

To clarify a desync is when the game simulation state on one machine is not a bit-exact match for another machine. So every machine has to process all simulation frames with a bit-perfect result independently of each other.

If Chobby is the cause this time, then publishing an older release of it should solve the problem and then the issue can be investigated further.

[keen dawn]

yeah it will be chobby 100%

it has to be

[proven inlet]

I don't believe it has ever been the cause in the past, but nothing is 100% certain where desyncs are concerned.

[strong thorn]

are more logs still needed? looking through the diffs on some of my local desyncs

only diffs here (take w/ grain of salt, idk this log format):

• thug unit has a movement command issued in one log and not the other causing slight positional/rotational differences

[keen dawn]

let's go rollback chobby PR

[amber wren]

ok, but i m at work and I cant do it via the github website

ok i teamviewed into my home and got it done

i forgot i could do this since i have tethered for a few months until recently 😄

[amber wren]

@peak prism just so you know why it's reverted for now

[peak prism]

just got home and trying to catch up real quick but gotta dip in a minute again
i'll be able to properly look how badly i screwed up in like 4 hours

[keen dawn]

Not sure if the change is live; however, for now no desyncs (since Server restart) (50mins in and no desyncs)

[peak prism]

server restart doesn't force chobby update, game restarts do
but if it was this that broke the game i panic

[keen dawn]

Well, can happen better now than after steam release o.O

oh just got desynced with updated chobby

[peak prism]

if this is it there will still probably be people on the old chobby version

[pallid bone]

Got desync, could it be related to the fact that I have my own gui_gridmenu.lua gridmenu_config.lua gridmenu_layouts.lua in data\LuaUI?

[amber wren]

those are widgets, they should not affect synced states

[keen dawn]

@amber wren desync fixed
@proven inlet

all lobbies where I told doggos to restart game have no desync since

7 games no desync

went through multiple lobbies - some had desync all all of them did not restart chobby (post update)

[proven inlet]

What was changed?

[keen dawn]

#🎲｜github-game message
Probably this

[proven inlet]

What caused the desync?

[keen dawn]

I am still checking lobbies for desyncs - they happen - but only to those who did not restart Chobby since update (or so is their response)

[amber wren]

Dunno

[latent citrus]

oh, so it is an issue. I got super desynced from my current game as well lol, I was getting a lot of frame-sync errors during the game

[keen dawn]

Another bunch of people desyncing pre-chobby; new chobby seems to not have any desync victims yet

Arrakis with forced new chobby/ restart - has 6th game without any desync

strait is on their 5th game

rotato 3rd game in a row clean

[willow totem]

how are you checking this? (geniunely curious)

[keen dawn]

I have spies

in each lobby

and checking manually as well

[peak prism]

I'm getting fired aren't i

[keen dawn]

But I should not be doing this - desync games were my best games

[keen dawn]

Don't worry lmao haha

since noone knows why this was happening I doubt ur to be blamed

Knowledge is a power; and it seems our power will be stronger after this o.O

[hoary whale]

Can that chobby stuff actually cause desyncs?

[keen dawn]

Well, does it make sense for you to get desynced in every game/ specate till you restart chobby?

I was spending hours checking what it could be inside of the actual game; however, then I was like - it has to be outside

haha

[candid berry]

Well, what if Chobby feeds synced configuration settings to the players and server, and when some kind of check occurs for those settings in synced space it causes desync.

But that's just a theory.

https://tenor.com/view/game-theory-game-theory-its-just-a-theory-game-theorists-gif-20883020

[peak prism]

-# just watch it check the synced state of the font color
-# i proabably shouldn't speak, i broke it

[hoary whale]

I feel like if we do solve it we should learn something from this. Like where to look for commits in the future. Seems like we weren't even looking at chobby at the start at all.

[keen dawn]

https://tenor.com/view/huge-hug-bromance-cute-dog-dog-love-love-you-gif-15055257

[keen dawn]

let's give it time - desyncs can still arrive

[peak prism]

cause we don't have a way to force people to update chobby unlike game version

[keen dawn]

One desync

There is no desync post update yet

[keen dawn]

Alright - first desync victim post update - Nalung

[keen dawn]

--- he might have it confused - on all games where he "was desynced" he actually was not.

-- checking stopped game (waiting for his replay)

roto lobby no desyncs yet (8th game)
Arrakis lobby 11 games no desyncs
Strait lobby 9 games no desync

[north sedge]

How on earth could PR #781 cause desyncs?

[keen dawn]

all lobbies no desyncs - only preupdate people

[peak prism]

-# it's cause am terminally stupid

[keen dawn]

nope; noone knows the reason - and as @proven inlet mentioned - this is basically first time happening

will be surely important for the future of bar dev

[peak prism]

there is a chance that some other desynces are also caused by bools and list options being able to hide other options

[north sedge]

How does Chobby even work? Does every player generate their own startscript for their own game?

[keen dawn]

He got it confused - he was not the one desyncing - the one desyncing was the usual suspect since post update

[keen dawn]

Usual suspect: EinBlatt - we were unable to get him to restart game o.O

kk safe to say - desync got fixed

[minor wasp]

Engine downloads the server start script on connection

AFAIK for multiplayer games anything inside of the local start script except for server host & port and player login & password doesn't matter

[keen dawn]

Some things to mention

1. Desync only happened to some individuals - over and over (the same usual suspects all time)
2. Skylobby users had no desync
3. When desync happened - it kept happening in all games till game restart - even as spectators
4. When multiple people got desynced - they played in their dimension together
5. Desync seemed to happen in between 8-12 min for all cases

[minor wasp]

The thing is that engine is not started 100% fresh for each game, it's "reloaded" while it's running. I don't know how throughout the reloading code is, never looked at that code, so theoretically, there might be some bad state preserved between runs or between lobby and game.

I've never heard about such issue in the past though, but who knows

[halcyon grove]

Maybe it doesnt cause it but just exposes it

[peak prism]

there is something interesting that i want to chase down, but would like a log or two more from desynced instances to see if it is related or simply a coincidence
laudy gave me gamestates but i want a log or two specifically
turns out it was just a coincidence. back to square one with me

[keen dawn]

Is this helpful?

I reproduced the desync 3 times around that time

[peak prism]

thanks

[keen dawn]

we got one desync post update from trusted source o.O

[peak prism]

log pls

wrestle it out of them if you have to

[keen dawn]

ah already had that in mind; but pm with you for now

Desync post update

@proven inlet gogo look into it if your eyes see something sus

[keen dawn]

hmm desync at 3:04ish
while zooming out

[keen dawn]

This is different desync and unusual to "normal" suspects - @proven inlet this might be engine related

[keen dawn]

might be connected to un-usual latency upon zooming out since engine update?

[willow totem]

Yeah i just desynced while spec-catching upto a game, Close BAR > Reopen > Rejoin and did not desync the 2nd time

[keen dawn]

do you use unitid widget by any chance?

[willow totem]

nope

[keen dawn]

this one

Seems like this widget is pumping out an unlimited amount of errors

#⭐｜main message
another desync

but they are much more rare now o.O

[keen dawn]

also relatively unusual desyncs (compared to old ones)

@proven inlet

[keen dawn]

@amber wren desync went absolutely mental

#⭐｜main message

full team desync o.O

[peak prism]

seems like it's time for floris to revert all of my chobby commits

[keen dawn]

perhaps you were not the cause o.O

it's so weird we had little to no desyncs and now we are overflowing again

[peak prism]

there is more similar code in chobby, what the pr floris reverted simply added one more instance of it being used

i'll take another look at chobby stuff tomorrow, and if i find nothing of intreast i'll make a pr to clean up all of my changes to the modoptions panel

[keen dawn]

Another desync - un-usual one (to the old ones)

[violet heron]

i think BAR got cursed

[keen dawn]

since teiserver restart/ robert revert we had hours of no desync and now we are back to square one

[peak prism]

would updating chobby reset some value stored somewhere?

[tulip silo]

desync in less than 10 seconds 2 players

[tulip silo]

can a error in teyserver cause desync?

i'm not asking you laudy, just brainstorming

[hoary whale]

Man we need a better way to do large rollbacks.

[amber wren]

i can confirm it still desyncs and indeed restarting chobby i can rejoin without sync errors

[tulip silo]

i can confirm, no desync with skylobby

[willow totem]

Have you had desync's using normal lobby?

[tulip silo]

i can try to log with chobby, is to much time that i do not use chobby, like 15 days, i will try to log in with no update

hmm, other system is fucked up(kernel panik 🙂 )

need to reinstall

the appimage

[reef kernel]

last game, had multiple desyncs for 2 players

[silk hedge]

We got desynced together, both from the same country, playing as a party

[tulip silo]

ah fuck appimage in 24.04 is fucked up

anyone here have see this(sorry offtopic)
The SUID sandbox helper binary was found, but is not configured correctly. Rather than run without sandboxing I'm aborting now. You need to make sure that /tmp/.mount_Beyondf8jWMA/chrome-sandbox is owned by root and has mode 4755.
Rilevato trace/breakpoint (core dump creato)
p@c47:~/Scaricati$

how many linux users desyncs?

[strong thorn]

Just curious, at this point, my understanding is we've reverted most of the PRs in the main repo, the only changesin the Chobby repo and there were no underlying engine releases around this time. What else could affect the code that's run?

Putting root causing aside for the moment, it seems like there isn't anything else to rollback? or am I missing something

[hoary whale]

AFAIK the legion commit has not been reverted. The order menu commit is one line that just checks for nil (prevents null exception error). I don't think it's a problem but who knows. There could be commits at or prior to 11 October that could be a problem though.

[astral phoenix]

Have we come full circle now, from the theory that it doesn't affect when legion is enabled to there is only a legion commit left?

[tulip silo]

@silk hedge @strong thorn can you please share me the replay with the last desync?

and you use chobby right?

[strong thorn]

yes

will do in ~ 20-30m after game

[silk hedge]

Chobby is the standard lobby?

Then yes, I do. The server replay: https://www.beyondallreason.info/replays?gameId=41d40e67af04caaf3f0f741d8bd5f5f7

[tulip silo]

in game name? and minute of desync circa?

[silk hedge]

Gumpert, party with asalways, first sync error at:

[t=00:02:03.782815][f=0035911] Sync error for Gumpert in frame 35907 (got e42ef8fb, correct is 689352e3)
Uploading the full log gives a discord error for some reason

[tulip silo]

okok

i see

is a desync fest

from the start moreless

i do not desync

with chobby

[tulip silo]

now i go to sleep

[strong thorn]

from one of my desyncs (and then from after I restarted chobby)

diff above if it makes your life easier: #1294721680379351150 message

[vapid pond]

Have you investigate in some "cache mechanism" ? Because it is weird we can temporary fix the desync with a full restart of the client. And i find the launching of the client weirdly slow recently with multiple cache process (previously it was 1 max/day)

[upper ivy]

half our lobby just desynced 10 min in. Idk how to upload logs but it was the game played by me just 5 min ago if that's able to be audited and is useful

[rustic viper]

I never had a single desync with Manjaro KDE (flatpak bar client), i also not using skylobby. But i see always other players with desyncs.

[tulip silo]

@strong thorn @silk hedge can any of you make a small test for me? 🙂

1 watch one replay where you desync and test if as spec you desync too(i hope that you will read into the error msg something like (spectator yourname desync)

then if this work

i will try to make you install skylobby and check if you desync with it, cause im not able to desync by myself

I can promise you no reward, but much gratitude

[silk hedge]

You mean I should watch my own replay of yesteday?

[tulip silo]

yes

theorically you have to desync 2 times

[silk hedge]

replaying my local copy does not show any desyncs, I've waited well past the first occurence in the log at t=00:02:03.782815

[tulip silo]

mmm interesting

and try to download the replay from the site?

is impossible that do not desync at all(or better it seem starnge to me)

[silk hedge]

So the server replay has lots of sync errors with a message like demo player 2 (slim[]daddy) does not match our checksum

[silk hedge]

Here is the log of the first part of the replay, lots of errors like:

[t=00:12:50.995414][f=0015064] Error: [DESYNC WARNING] checksum a14b0ca6 from demo player 5 (asalways) does not match our checksum 7ee14edf for frame-number 15060
[t=00:12:50.995490][f=0015064] Error: [DESYNC WARNING] checksum a14b0ca6 from demo player 1 (Gumpert) does not match our checksum 7ee14edf for frame-number 15060
[t=00:12:51.020121][f=0015067] Error: [DESYNC WARNING] checksum a14b0ca6 from demo player 2 (slim[]daddy) does not match our checksum 7ee14edf for frame-number 15060

[tulip silo]

yes but you as spec are not desync

[strong thorn]

@tulip silo , I looked at a couple of other desyncs posted in this channel and can confirm I get desyncs as spec. also on a site note, also a dev just not for bar b/c i hate lua actually not sure, might take another look. Don't think I got desynced as spec, just saw the desyncs happen. I wasn't able to repro the desynced player's pov from chobby

[upper ivy]

dang so still no fix for this? We can't do a rollback to a previous version?

[hoary whale]

They have done a big revert. Problem is you need to host new lobby to get latest version. I recommend doing a restart server but the people who can are not in.

[dim flower]

We've done a rollback yesterday to supposedly last version that had no desyncs

Welp... as jauggy said, unsure if the logs that keep coming are from post-revert or not because of how the log collector works.

[vapid pond]

No desync since 10h of gaming

[violet heron]

Anyone knows what caused them a d whos fault was it?

[tulip silo]

We are dig, problem is not properly fixed, we reverted a big amount of pr.
Hope this fix in the meantime we dig into the sand

[strong thorn]

fwiw rollback seems to have fixed

hurrah it is not chobby

[hoary whale]

Yeah but then we have to add commits one by one later to figure out which one causes it.

[strong thorn]

yeah binary search

[amber wren]

only encountered 2 desync cases (single people) when i rejoined 28 games yesterday

[proven inlet]

We may need to see the stats from Beherith once we've had a day or two, to check whether the desync rate is back to nominal levels.

[gloomy tapir]

Still having desync issues XD

[dim flower]

gg

We're fucked

[gloomy tapir]

I'll just take a break and focus on other hobbies lol

[dim flower]

🫡

[hoary whale]

I have not had any desync issues lately.

[willow totem]

fwiw I've not seen any since the large rollback (and a few hours after while people and lobbies updated)

[violet heron]

do we know who caused it?

or is it still mystery?

[minor wasp]

It doesn't matter

[violet heron]

well i kinda thing knowing why it happened is important
so not to happen again

[minor wasp]

Why it happened is important, who triggered bug is not

[violet heron]

it is
we need to know who to sacrifice to volcano

[peak prism]

-# i'm always available to take blame, i don't contribute anything anyway

[violet heron]

https://tenor.com/view/gru-despicable-me-explode-gif-18746054

[peak prism]

but no, figuring out why and what caused the desync is more important than who caused it,
so that we can fix the underlying issue, or at the very least prevent it in the future

[minor wasp]

I volunteer you

[peak prism]

i can't belive Aqua would ruin bar, i thought we waterproofed the servers

[violet heron]

im not a dev so it wont work

[minor wasp]

well we can't sacrifice to volcano people that can fix it

[candid berry]

[violet heron]

but we need to know whos the evil dev

i need to know who to blame for losses

cause otherwhise its my fault

and i cant blame myself

[peak prism]

there is no evil dev,
this didn't happen on purpose

[vapid pond]

Always no desync, look good guys

[violet heron]

sounds like something a evil dev would say

[tulip silo]

This seems like something a evil player would say

[dawn scaffold]

I think it’s time to close this thread

[violet heron]

https://tenor.com/view/cfif126-cfif-gif-20981448

[alpine briar]

Investigations ongoing, please do not blame anyone.

Situation stable for now, pending more accurate reporting measures.

[brittle parrot]

Yeah, everyone makes mistakes, and frankly totally avoiding bugs while making a game is an impossible standard. No reason to scapegoat anyone. Only thing you can do is try to learn from the incident to minimize future issues. And getting rid of someone (or discouraging their continued help) who has learned the lesson for someone who hasn't yet learned the lesson is usually unwise.