Questions regarding synchronization in Netcode for Entities [1.0.12]

[cursive anvil]

Hey there friends 👋

first off, I'd like to mention that I've read the documentation twice now and I dug into the ECS samples for Netcode for Entities. But for someone new to Netcode for Entities it is still difficult to wrap my head around some of the concepts. In general, a more problem-oriented approach for the docs would be nice, e.g. "use this concept to solve these types of problems, and for these other types of problems do this instead". The code in the samples also lacks comments, making it difficult to figure out why certain design decisions were made, and whether they were necessary or a choice. I hope you consider this constructive criticism, as I've been trying to learn this framework for a couple weeks now and still feel like I barely understand the basic concepts. Every time I read the docs, I feel like half the stuff is new again 😅

That said, I have some concrete questions, I hope someone might be able to help me with.

1. The docs say one needs to use the GhostFieldAttribute in order to sync components on a Ghost entity. In the samples, this is for example used for the ServerHitMarker in the HelloNetcode HitScanWeapon sample. The ApplyHitMarkSystem of the same sample updates inside the PredictedSimulationSystemGroup and uses IsFirstTimeFullyPredictingTick to abort execution when rolling back. But I'm not entirely sure why or when this check is needed (see next question). And why even check for the first time a tick is predicted? What if the prediction is wrong? Then whatever executed can't be rolled back, right? So wouldn't it make more sense to check for the tick just before the first fully predicted tick? I think a diagram about the exact order of all the things happening in Netcode for Entites would go a long way for new people to better understand it.

2. What adds to my confusion is that in the Respawning sample, the Health component e.g. does not use any GhostFieldAttributes. On top of that, the DamageSystem (which also updates inside PredictedSimulationSystemGroup) does not check for IsFirstTimeFullyPredictingTick. So if checking IsFirstTimeFullyPredictingTick is required in PredictedSimulationSystemGroup to prevent things from executing too often during rollbacks, then why isn't that required for deducting Health as well? And how is it even synced to the client without the GhostFieldAttribute? How does it manage to be server authoritative in this case?

3. When do I need to work with NetworkTime and ticks and all that, and when can I just use SystemAPI.Time.DeltaTime?

4. One of the things I would like to do is make slight (client authoritative) adjustments to the LocalTransform of Ghosts in the client world. What would be the best way of doing that? Have a system run in the PresentationSystemGroup that changes the LocalTransform on those Ghosts?
   .

5. The reason I'm asking 4. is, in the beginning of my game, I'd like to animate setting up the playing field. And I would like to sync that animation with the music playing locally on the client. Since those objects I want to animate are physics objects that can be destroyed later on during the game, I need those objects to be ghosts. However, during the setup, I need their position (at least visually) to be determined by the client to achieve near perfect sync with the client's music, on each individual client. @hardy thicket touched on that a little bit in this thread https://discord.com/channels/489222168727519232/1123328909417660436 but the problem is when it comes to music playing on the client, the audioSource's time or the dspTime needs to be the ground truth. Any attempt of adjusting the audio results in horrible artifacts. And for some use cases, being even 50ms out of sync is unacceptable. Due to slight lag spikes the audio's time could diverge from the application's time at any moment, so to achieve perfect sync, when trying to sync to a longer audio track, a custom clock that is constantly and smoothly adjusted based on the audio's time is needed. From what I can tell, Netcode for Entities does not offer an obvious way to accommodate this. What I would imagine is something like a DeviateableGhostFieldAttribute where the server actually keeps track of two values of the same type, one purely server authoritative, and one that is fed back to the server by the client. And when setting this attribute, the developer could set a threshold value of how much divergence is allowed before the server no longer accepts the value. As long as the delta between client-value and server-value is below that threshold, server will accept the client value without demanding a rollback, but as soon as the delta is greater, the sever makes the client reset (if lag gets too big then music's time has to be adjusted afterall, but way less often than without such a special field). Just a thought of course. Given my limited understanding of Netcode for Entities there might be other or better solutions to allow some things to be client authoritative to a certain degree.

Sorry for the wall of text, but I really feel like I'm at the point where I need some pointers to get any further in my studies of the Netcode framework. I'm thankful to anyone who might have some insights.

[hardy thicket]

Hey CG:

Q1:

• On the client, a hitmarker is essentially a "client predicted event". I.e. You only want to spawn this hitmarker's UI once, for your first prediction. If you rollback after predicting the hitmarker, and predict the hitmarker again (possibly in a slightly different location), you'll end up with two hitmarker SFX GameObjects, both corresponding to one shot. This is why this system is guarded by IsFirstTimeFullyPredictingTick. On bad pings, you may predict the same tick ~20+ times, leading to 20+ hitmarker UI GameObjects (spites) for 1 hit. Thus, IsFirstTimeFullyPredictingTick is a way to guard events.

Note that the writes to these values aren't read by the rest of the simulation, thus this is safe. You can only do this for stuff like events, as otherwise it'll break rollback + re-prediction.

• You only know that it was a miss-prediction once the server sends a snapshot confirming you were wrong, which happens much later. This is the trade-off you make when you use client prediction altogether: It's unreliable, and could be wrong. But because we (the game designers) have decided to predict hitmarkers, we accept this trade-off. Some games use hitmarkers to show "yes the server confirmed this hit". In other games, hitmarkers are "gameplay feel" (a.k.a. juice), and are therefore client predicted (as they're rarely wrong). In our sample, we show both the predicted marker, and the server confirmed marker.
• Note: The server ignores this flag, as it's always true on the server (as the server doesn't rollback).

Q2.A: the Health component e.g. does not use any GhostFieldAttributes
This is because your opponents health is entirely Client Predicted, so it's never synced. You can see this in the Healthbar sample, if you trigger a 10s lag spike and then shoot the opponent, their health will go down, even when you're not communicating with the server. There is actually a bug where shooting someone to 0 health (but they don't actually die) will despawn the healthbar, when it shouldn't be.

Note: In a real game, you would want to make HP a GhostField (so you can see damage inflicted by other players). I'm not sure if our sample resolves this by predicting other shots too, but it's "wrong" for a real game use-case. We should probably fix that.

What is NOT predicted is "enemy death". You can see this via the [GhostField] public bool Enabled; on AutoCommandTarget, which is the proxy we use to determine the "dying" state. Obviously, on death, the player will entirely respawn, and ghost respawns are replicated automatically.

Q2.B: the DamageSystem (which also updates inside PredictedSimulationSystemGroup) does not check for IsFirstTimeFullyPredictingTick

• Dealing damage predictively is different to the above HitMarker case because it should be "continuous" (i.e. not an event). Therefore, when we rollback and resimulate, we resimulate the damage dealt. This is why it is not guarded by IsFirstTimeFullyPredictingTick.

EDIT: I've just seen that ShootingSystem does in fact guard by IsFirstTimeFullyPredictingTick. This sample was built exclusively to show Hit events, it will not work if you attempt to modify it to include [GhostField] public float CurrentHitPoints. I'd recommend writing that yourself for now.

Q3: When do I need to work with NetworkTime and ticks and all that, and when can I just use SystemAPI.Time.DeltaTime?
Depends on the use-case. I'd recommend viewing more of the samples (e.g. https://github.com/Unity-Technologies/MegacityMultiplayer) for "game" use-cases.
Most of the time, you can just use DeltaTime. IsFirstTimeFullyPredictingTick is more for events and one-shot use-cases. But networked games involve a lot of this kind of nuance, so it's hard to avoid in practice.

Q4: One of the things I would like to do is make slight (client authoritative) adjustments to the LocalTransform of Ghosts in the client world. What would be the best way of doing that? Have a system run in the PresentationSystemGroup that changes the LocalTransform on those Ghosts?

You'll need some kind of server-authoritative game-state enum saying "clients, show the pre-game animations right now". Use that to trigger the client-driven animation. But the server IS the authority on when the game actually "starts". You could have a system that waits for each client to "confirm ready", but then you'd also need a timeout server authoritative override.

[hardy thicket]

when it comes to music playing on the client, the audioSource's time or the dspTime needs to be the ground truth. Any attempt of adjusting the audio results in horrible artifacts.
Due to slight lag spikes the audio's time could diverge from the application's time at any moment, so to achieve perfect sync, when trying to sync to a longer audio track, a custom clock that is constantly and smoothly adjusted based on the audio's time is needed.

There are two kinds of use-cases I can think of here.

For the FIFA "match warmup announcement" use-cases (similar to a Mario Kart style "race track overview intro"): You can do something much simpler.

• The server communicates what ServerTick this animation started, and how long it'll play for.
• The clients start their animations smoothly the moment they see this GhostField value.
• Late joining clients can either start their animation half-way through (the animation + audio should allow for this), or start from the beginning, but with a countdown saying "game start in X". I.e. It should be interuptable. If you have a high enough production value, you could even have a very short version of the animation that you can play for late joiners (e.g. "This team just about make it onto the pitch for the whistle! Let's get to it!"), with some extra affordance in the kickoff state.

For the Call of Duty "getting out of the helicopter" use-cases: It's a little harder, but still doable.

1. Wait for all players to have confirmed ready (with timeout affordances).
2. Trigger an animation to start ~1s in the future of a given ServerTick (to guarantee all clients start it on their own client timeline).
3. "Client Predict" the animation. Perform no correction. Even predict the client taking control of the character when the animation ends.
4. Handle desyncs once the player is actually running.

This is what happens when you get it horribly wrong lol: https://www.youtube.com/watch?v=f5fwgHpUR1w

Important learnings here:

• Even if the animation is client-side, the server still needs to KNOW where to spawn the ghost. It looks like Halo: Infinite used a hybrid approach? Animation clientside, root position serverside. But they managed this syncing incorrectly, thus players facing the wrong way.
• If the animation allows players to end up in different locations (where the animation is anchored to a map spawn point), you'll need to send the final client position as a one-off input command (or even RPC). You should do this as early as possible to give the server time to take that input and update ghosts with it. Note: This is trusting the client! Clients could exploit this with hacked coords!

As for how to actually override the client animation: We don't currently have a mechanism to briefly disable syncing of a GhostField or ghost (except for relevancy, but that'll despawn the entire ghost on the client).
BUT, you probably don't even need to. As you said, you only need to override position (I.e. LocalToWorld in this case) + animation state. That should be doable.