bevy_replicon

[spring raptor]

Will make them pub with and address the review suggestions today, will ping once I done.

Or you can make a PR to my PR and just mark everything you need as pub.

Only if you need it right now. If not, I will do it in a few hours.

[echo lion]

@spring raptor I don't understand what's the point of having different serialization functions per replication rule. In this PR replication rules are essentially just 'only replicate if these other components are present' or more broadly 'replicate if any group you're in intersects with the entity'.

At most having multiple serialization functions let you crop out parts of a component that are custom-default-initialized per group.

[echo lion]

I think it would make more sense for command fns to be tied to replication rules, and then within command fns you can branch on different existing client components. And if necessary, insert a 'temporary' value while waiting for blueprints to be applied.

[dire aurora]

ReplicationRules are how a component/group got registered, so they wouldn't really be related to commands fns which are per component? 🤔

[echo lion]

The idea is you select command fns based on client archetype, which is derived from server archetype, which is represented by the groupings in replication rules.

So they are related.

[dire aurora]

I don't think I see any changes on this PR to how the rules get picked, you have your serialization/deserialization rules per components, whichever is the highest priority (most likely from a custom group) gets used. That way you can have standard implementations for components, and make different optimized implementation for niche cacses where parts of a component become irrelevant (If your characters two of their rotational axes locked, you don't need to send a full Quat nor a full angular velocity)

[echo lion]

My point is now replication rules only do serialization stuff, without any direct impact on what writing logic is used.

Like yes, serialization optimization is nice, but it's a pretty weak use-case for such a large feature.

[vapid badge]

I'm pretty new to rust/bevy/replicon so I don't know if I'm doing something super crazy here. I wanted to have a sorta-clean way to represent "Blueprints" that will get created on the client side, and I came up with something like this (with some hopefully inferrable hunks elided):

pub fn blueprint_dispatch<T: BlueprintLabel + Component>(
    mut commands: Commands,
    registry: Res<BlueprintRegistry>,
    q_blueprint: Query<&T, Without<BlueprintProcessed>>,
) {
    for b in q_blueprint.iter() {
        if let Some(system_id) = registry.get_blueprint(b.intern()) {
            commands.run_system(system_id);
        };
    }
}

pub trait AppBlueprintExt {
    fn blueprint<C>(&mut self) -> &mut Self
    where
        C: BlueprintLabel + Component;
}

impl AppBlueprintExt for App {
    fn blueprint<C>(&mut self) -> &mut Self
    where
        C: BlueprintLabel + Component,
    {
        self.add_systems(
            PreUpdate,
            (blueprint_dispatch::<C>,).after(ClientSet::Receive),
        )
    }
}

Are there existing ideas in this space I should be checking out?

[echo lion]

Generally blueprints are entity-specific, but in this code it looks like you're treating them as messages.

[vapid badge]

Good point, I do need to get the entity from q_blueprint and then use an input param on the one shot system maybe?

[echo lion]

Yes that's probably what you want. You could also just write systems that check for Added (although there are perf implications that your solution might improve depending on the case).

[dire aurora]

The serialization optimization was a pretty small part of the previous PR. The current PR is about separating writing, which is done based on logic only the client knows, from deserialization, which is dictated by the server ... Writing can't be chosen by the server, since a server entity will have many different client represenations

[dire aurora]

Yea, you'd definitely need to pass in the entity, or you'd need to set some resource to the data the system would need (the latter only makes sense if it clutters up every system with some optional data)

[spring raptor]

I will quote you from GitHub to add more context.

The problem is it can't be guessed until after the first tick of replication, because CommandMarkers are filtered based on the client archetype. So you always have one tick of ambiguity.
This is correct. When you deserialize an entity for the first time, you don't have any marker on it, you insert them after spawn. But it's also fine for prediction and interpolation for already spawned entity to insert components directly. Am I right, @dire aurora?

But I see how this could be a problem for you if you want to just customize writing and reading. This is why I suggested to allow overwriting default write functions (the one that used without any marker). Will this work for your?

we know the replication rule FnsId, which represents a segment of the server's archetype.
Yes, but server doesn't have any prediction or interpolation components. This is why we can't rely on server to pick write function. This is the reason for the separation and the whole marker API.

[vapid badge]

The oneshot system can query for a marker enum / component of some sort but then I guess I'm just adding a bunch of extra steps for the same result.

I was trying to avoid having a giant enum of blueprints types somewhere but I think I might be adding complexity with little benefit. I'll experiment with an In() param and then actually implement this for my entities instead of just a unit test and see how it feels

[echo lion]

The fundamental problem is when an entity is first replicated, the client has no information for writing. So you are always off-by-one-tick.

[dire aurora]

I think in pretty much every case the first thing is fine, predicted entities already have their marker before replicon even writes to them, all other entities need to get spawned before the user's system can decide on what to add to the entity, and in most cases you don't want an entity to only have Remote<T> but no T 🤔

[echo lion]

I think each FnId should get its own customizable write per-component that falls back to a global write, and then you can further override with client-side markers. This would be most flexible.

[dire aurora]

Hmmm, what would be put in a per-component write? 🤔

[echo lion]

So it would be: highest priority client-side marker -> FnId custom -> global custom -> global default.

[echo lion]

Same thing you'd put in a client-based write except using replication rule groups to select instead of client markers.

I'm thinking beyond prediction/etc to the fully generalized API.

[spring raptor]

We can do it, but this way registration of rules will be unsafe :(
It's because there is no way to check if the passed write fn can be called with this component...

Do you need such an advanced customization?

Under "overwrite default write" I mean to overwrite it per-component. I.e. if you have a ComponentA, you will be able to set the default write for it. This way API will be simple and safe.

[echo lion]

Hmm let me think

[dire aurora]

The unsafe part is the custom global one right?

[spring raptor]

I talking about this write:
https://github.com/projectharmonia/bevy_replicon/blob/1455ab406f1475e043d28221e1e26f779a3db5ae/src/core/replication_fns/command_fns.rs#L22
It's hardcoded here:
https://github.com/projectharmonia/bevy_replicon/blob/1455ab406f1475e043d28221e1e26f779a3db5ae/src/core/replication_fns/command_fns.rs#L32
And it's not-overridable.
I proposing to add a simple setter.

And a function to the extension trait for App for simplicity.

[spring raptor]

Yeah, the same reason why register_marker_fns is unsafe. We can't validate that passed C is the same that used inside write.

[dire aurora]

Yea but it would be worse with the custom global, since we wouldn't be able to instantiate a C variant of that function

[spring raptor]

Sorry, I probably confused you. I shouldn't call it "global". I meant to overwrite the default write for a component.
See this message for details: #1090432346907492443 message

[dire aurora]

Ah yea, if we do it per component it could at least be registered in a somewhat sane method that prevents you from passing in a write::<B> for A

[echo lion]

Ok I think for now we can just allow overriding the component's default fns. If a use-case crops up for rule-based overrides we can revisit.

[spring raptor]

Our write is generic, but user's write can be any function, the signature has no generics.
It's the same story as with register_marker_fns. Despite we pass C to the function, we can't forbid user to call write::<B> :(

[spring raptor]

Yeah, with the default override per-component it will be the same functionality as in the previous release.

[dire aurora]

Oh I think I see what you mean ... Can't we just accept a version of the function that still has a type argument, then transmute that away internally? 🤔

[echo lion]

One way could be to have the type ID as part of the write function signature, store the component id with the write function, then use an orchestrator function to pass deserialization callbacks to the transmuted write function.

fn orchestrator<C: Component>(component_id: ComponentId, serde: SerdeFns, cmds: CommandFns, cursor: ...) {
    assert_eq!(component_id, serde.id);
    assert_eq!(component_id, cmds.id);
    let deserialize = |...| serde.deserialize::<C>(...);
    cmds.write::<C>(..., deserialize);
}

Then write doesn't need to know about SerdeFns or Cursor.

[spring raptor]

It might work if I combine both of your suggestions 🤔

[spring raptor]

Wait, user still will be able to put any C.
Like deserialize_transform<C> and it will be unsafe.

[dire aurora]

Calling deserialize then write has some pretty annoying downsides, tho we already do this typecheck when you call the functions trough SerdeFns I think?

[echo lion]

A callback is passed in, you decide where the callback runs

[dire aurora]

We would be passing in write functions, most of them would already have a generic argument ... I'm just not sure if rust's type system allows us to pass it around as a still generic type 🤔

[spring raptor]

Most of them, yes, but technically the following will be unsound: deserialize_transform<Visibility>.

Yes, it's silly, but it will be possible to trigger UB with only safe code.

[dire aurora]

Hmmm, actually we wouldn't be able to take a generic write function ... It doesn't have any type info in it's signature 🤔

[spring raptor]

Yeah, we can put it, but it will be "dummy".

Ah, and instead of deserialize_transform I wanted to write write_transform.

[dire aurora]

Yea if you make a specific version for write_transform and pass in a different type that would still cause problems

I guess there's two issues here: Passing in A as type argument, then giving functions for B. Or passing in an unsound write function, that insists the type is Visibility when it's not ... I think we do have assertions to catch both, but the first is easy to mess up if you set your own write functions 🤔

[spring raptor]

While it will be safer, it will add some overhead via non-inlinable call and runtime type check.
I think that safety usually worth it, but this one called in the hottest path...

[spring raptor]

And I do only debug assertions for types.

[dire aurora]

We also already do these assertions

[echo lion]

The type check would just be two equality checks, not that much. I'm not sure how impactful not inlining would be. Is deserialization in the PR actually inlined? Since you are passing in function ptrs.

[spring raptor]

Ah, right, we do it in deserialize. But it's debug assertions.
I can turn them into regular ones, this way it will be safe. Just not sure about the cost, maybe it's minor?

[echo lion]

I think using callbacks would just add one ptr indirection.

[spring raptor]

Yes, deserialization is not inlined too. But this way it will be one more call.
But it also improves the API a lot... 🤔

[dire aurora]

The check is cheap, the branch is expensive. It's pretty much the same thing as why bevy is littered with debug_checked_unwraps

[spring raptor]

Right

[spring raptor]

Do we actually need these asserts?
You suggestion adds C to write and looks like it solves the problem?

[dire aurora]

Passing C to write isn't possible outside of the transmute route

Which isn't necessary since the function's body already knows what C is

Tho maybe the deserialize signature would have it? 🤔

[spring raptor]

Yes, but Koe suggests to add one more function that will do it for us

[dire aurora]

And also it would need to be a type that has deserialize and deserialize in place, we don't know which the write function needs after all

[echo lion]

As long as you assert somewhere

[dire aurora]

But that should be pretty minor, just two callbacks in a struct

[spring raptor]

I think this will work by the cost of adding one extra function call. And public API will be safe.

CommandFns will be similar to DeserializeFns with erased pointers.

But let me try it in a separate PR after merging this one.
I think that the suggestion is great, but it will require some experimentation.

[dire aurora]

Also those examples in comments suggesting to make a per-type marker should probably be changed to use a generic marker

Cause that per-type marker would probably get expensive really quickly, and people might think this is the intended way to use the API 🤔

[spring raptor]

Yeah, I also realized it, but forgot to change, thanks

Going to address this and all other review suggestions soon

(within a few hours, will be away)

[spring raptor]

Quick question. Maybe it will be better to run an actual simulation? We provide a convenient module for it: https://github.com/projectharmonia/bevy_replicon/blob/master/src/test_app.rs
Otherwise you basically test replicon setters instead of your functions...

Usage example: https://github.com/projectharmonia/bevy_replicon/blob/3cbbfa9244b12f73d3408e7a9afd038a51b321c9/tests/insertion.rs#L8

[dire aurora]

If I get the values I put in the rule, then run those functions, I would be testing if I my rules are correct and if the functions I provided are correct no? 🤔

[spring raptor]

Ah, you also want to call them. Okay then.
But write isn't very convenient to call: https://github.com/projectharmonia/bevy_replicon/blob/ffe51df8d98b585fcec0ec53fb83b986917587c3/src/core/replication_fns/command_fns.rs?plain=1#L118-L127
Are you sure that you don't want to try test_app instead?

[dire aurora]

I need to test serialization functions so those are a bit easier to call (besides the unsafe ofc)

[spring raptor]

Done: https://github.com/projectharmonia/bevy_replicon/pull/227/commits/c5a159782a8d569199b60eca8b077519f82d2667
Let me know if I missed anything.

[rotund vine]

Hi, I'm trying to spawn an entity on the server and have it replicated to clients

But it doesn't get replicated

What could be going wrong?

The entity has the Replication component and components that are supposed to be replicated

solved it

great crate thanks

i don't understand has_authority

here it's used as "client and singleplayer" but elsewhere "server and singleplayer"

[dire aurora]

Pretty sure that's a mistake, has_authority is only true for server and singleplayer

[rotund vine]

oh that's what i expected

is not(server_running) the intended condition there?

[dire aurora]

Hmmm, I think some of these setters could stay pub(crate) or pub(super), and maybe that components field too (the old getter that returned &[FnsId] would work fine for any tests) ... The component_id and fns_id getters on FnsInfo need to be pub too, after that my test work ... And now that my tests pass that means I have bundlication mostly working with replicon's new API ... Which means it's time to start refactoring my game to use replicon

[dire aurora]

I think the run_if is actually supposed to be on send_events

[rotund vine]

replicon won't replicate from clients to the server right?

[dire aurora]

Yea it's only server -> client, otherwise clients could cheat in pretty much every possible way

[spring raptor]

Yeah, it's a mistake. It's even fixed, I just didn't release a new version.

[spring raptor]

What setters do you think I need to hide?

I made fields public because it's actually impossible to screw up. Once you insert ReplicationRule into resource, you will be able to only get a reference. Before insertion user can do anything with this struct.
Same about CommandMarker.

[dire aurora]

Ah right ... It's just the insert methods that might be a bit weird then 🤔

[spring raptor]

You mean like CommandMarkers::insert?

I.e. things that you can also insert via App.

[dire aurora]

Yea, it's there as a resource so people might thing they can get a ResMut and insert things, which is obviously not how it's supposed to work

[spring raptor]

It will work if user insert it this way, but I agree that it's not the indented usage, will hide.

[rotund vine]

hiii

i've tried replicon_snap but it doesn't work as well as i expected (probably because i use bevy_tnua as well)

i think i want to roll my own, any pointers on how?

currently planning on just running the character controller on both the client and the server

[spring raptor]

I think @dire aurora is already working on it.

[spring raptor]

I also just shared a link with a bunch of articles on this subject: #networking message
For replicon you just need to override how component is written and execute your custom logic.

[rotund vine]

thanks

[spring raptor]

Addressed: https://github.com/projectharmonia/bevy_replicon/pull/227/commits/6d9e7257c83d4e0d6c9f86e3f8f5830d62ebc8b9

I also addressed all@echo lion suggestions, except the one about CommandMarkerId. I left a comment https://github.com/projectharmonia/bevy_replicon/pull/227#discussion_r1571336593
Just not sure how to proceed. Maybe it's better to use usize to avoid the mentioned confusion.

[dire aurora]

Not interpolation, tho I'd imagine implementing that should become a lot easier with the write functions, which would also mean if any crate doing so has issues other people have some actual chance of understanding how it works

[dire aurora]

CommandMarkerId isn't public either afaict, so it shouldn't be a major concern ... I guess the confusion here comes from calling it "Id" instead of a more accurate name like "Index" 🤔

[spring raptor]

Yeah... I used Id because in Bevy ComponentId is also an index actually.
But I guess hiding this implementation in Bevy is fine.
Probably in case of the marker it's better to rename it into MarkerIndex (CommandMarkerIndex is a bit long?) to clarify that it could be invalidated after next insertion.

[dire aurora]

Is there anything in replicon to register player entities for each client?

[echo lion]

There is not. What behavior are you thinking of?

[rotund vine]

i just spawn them on connection

[dire aurora]

Something fairly simple, pretty much just a map from ClientId to Entity, the Entity only being there if the user registered it of course ... Not something every game would need, but in the decent chunk of games where a client only has 1 entity it would make plugins for simple tasks like input queueing a lot simpler

[spring raptor]

Another big PR merged 🎉
Still need to implement a history switch and experiment with write safety + add context + inclide tick into serialization.

Will start working on it soon, got sick today :(

[spring raptor]

Maybe keep track of such entities on the plugin side for now and see if it causes you any troubles.

[echo lion]

That just sounds like 'spawn entity on connect', pretty simple for users to implement on their own.

[dire aurora]

I can't really track them on the plugin side (since those plugins aren't the ones spawning the entities), but for now I just have a map in each plugin that doesn't already depend on one with an existing map ... I currently have 2 but I should have 3 so I might've messed something up somewhere 🙃

[spring raptor]

Not sure if I understand you. Do you have multiple plugins?

[dire aurora]

I mean every app has multiple plugins, but I have a few shared crates in my project that I might split out to a crate I'll publish later, so they don't depend on the rest of the game

Thus each of those has their own map

[spring raptor]

Ah, got it. What functionality are you planning to export into crates?

[dire aurora]

Atm my collection of crates that are like that are:

1. A crate of ID'd assets (tho this one might be less necessary since leafwing_manifest might have similar features?)
2. A crate with some evil macros for enums bitmasks
3. A crate with input queue logic
4. A crate to synchronize clocks (tho a lot of code for this one actually isn't inside the crate, will have to fix that)
5. Rollback
6. SDF collisions
7. SDF renderer
   3, 4 and 5 would be the useful ones that other replicon projects might use ... 1 and 2 could be useful for a few more networked games ... I have a few more things that would make sense to separate out except I know there's bevy features in the works that would make them obsolete 😂

[rotund vine]

3 and 5 is essentially what i need

i was thinking of doing a frankenstein thing from lightyear's prediction/interpolation stuff but moved out into a seperate crate to be used with replicon

i really like replicon's API but lightyear's prediction/interpolation seems (haven't actually tried lightyear) mature

[spring raptor]

That's a lot of crates 😅
3 and 5 have their own maps? I was going to suggest to combine crates that need the map into a single map, but these two not very related...

[dire aurora]

It would be 3 and 4, I might be able to avoid it with 4 tho 🤔

[spring raptor]

Thanks :)
We don't have prediction/interpolation because not all games need it (like mine). I want to maintain things I use, and let other people maintain things they use. It's healthier for the ecosystem, I believe that modularity is the key.

[rotund vine]

yea I understood that, I meant stuff like timewarp+replicon or replicon_snap

like "the replicon ""ecosystem"""

[dire aurora]

This is why I find it kind of scary that every new user seems to pick up lightyear

[spring raptor]

I think that the main reason is that we don't have a mature solution for rollback and input buffering.
In my opinion we have a nicer API.

[dire aurora]

The replicon API definitely feels pretty easy to use ... With bundlication I sort of knew how everything worked because I made it myself, but with replicon a lot of things make sense out of the box

Not having to make those stupid bundles with 1 item in them that doesn't even use any bundlication properties is also nice 😂

[spring raptor]

And I like that now we have a flexible combination of bundles and components. Because my old solution with only component-based rules was quite limited.

[dire aurora]

Yea it's pretty nice ... I ported all my replication rules pretty easily with the new system ... Porting events was more effort ... Sadly we don't have visibility layers yet, I have 3 places I use those, for now I just send the data to everyone, even if it makes no sense ... Still need to do all the logic for spawning entities and matching them between server and client tho

[spring raptor]

Cool!
You mean visibility layers for events?

[dire aurora]

No, for components ... It would be a good bandwidth optimization if we have a solid system for it, tho nothing too critical for development

[spring raptor]

Ah, yes, definitely a needed thing.

[dire aurora]

I just checked and I register surprisingly few things ...
I call .replicate 4 times, .replicate_group 12 times, .add_client_event 4 times, .add_mapped_client_event 1 time, .add_server_event 3 times, and .add_mapped_server_event 1 time

[dire aurora]

I set my apps up like this:

1. Register RepliconCorePlugins
2. Register all plugins that call replicate/add_x_event
3. Register the transport plugin, the respective (client on client, server on server) replicon_renet and replicon plugins
4. At the end of app init or during runtime I insert the transport and client/server with Replicon's channels
   I think that should be correct since the channels match and nothing crashes anymore ... But sending events doesn't seem to work and I have no clue how to debug this 🤔

[spring raptor]

So you have two different apps, one for client and one for the server?

I think that we need to add trace! for events. We have it for replication, worth to do the same for events.

[spring raptor]

On unrelated note, I feeling better today and just opened a PR with the discussed improvements for API safety: https://github.com/projectharmonia/bevy_replicon/pull/233

Going to look at trace! soon.

[dire aurora]

Yea

Hmmm ... Only thing I'm seeing when I enable trace logging is the debug messages that now pop up about a replicate::<T> being ignored because it's also in a bundle ... Should I be seeing anything about clients connecting or whatever? 🤔

[spring raptor]

We don't trace connections, but you could try to enable it for renet, it should show you information about it.
Maybe it worth to print it on replicon side as well....

[spring raptor]

Ah, you should definitely see messages about replication. It's under trace!. If you not seeing it, then you aren't connected.

I will be away for a few hours, when I back I will add trace! for events.

If you need it right now, you can add it yourself to client_event and server_event modules, it should be easy.

[dire aurora]

Server has no clients, which makes sense cause the client never sends anything ... I guess replicon is somehow not aware that the client is connected 🤔

Also adding some debug prints to my transport reminded me of this horrible Vec<Payload> renet has

Apparantly renet added connection states back and expects the transport to manage it ... calling .set_connected() in the client transport plugin was literally the only thing I had to do ...

Huh ... Why does replicon send 4 small packets, and why does renet not group them? 🤔

2024-04-22T18:55:13.825575Z  WARN transport::server: Received 21 bytes from ClientId(15)
2024-04-22T18:55:13.825586Z  WARN transport::server: Received 21 bytes from ClientId(15)
2024-04-22T18:55:13.825591Z  WARN transport::server: Received 27 bytes from ClientId(15)
2024-04-22T18:55:13.825597Z  WARN transport::server: Received 7 bytes from ClientId(15)

[spring raptor]

Ah... Are you using your custom transport?

[spring raptor]

Hm... In replicon we manually group update messages up to packet size, but events use different channels, this is why you may see many messages. But shouldn't renet group them?

[dire aurora]

Yea

Having some issues with disconnecting ... I remove the transport and RenetClient when I disconnect, which should cause the run conditions to trigger, but they don't ... So replicon doesn't see the disconnect, and also doesn't see the connect later ... It just goes to connecting because resource_added does trigger ...

[spring raptor]

Hm... It should work, we track RenetClient removals here: https://github.com/projectharmonia/bevy_replicon/blob/577178f783cd82fdc297a5ef6faaca2edd975243/bevy_replicon_renet/src/lib.rs?plain=1#L176

[spring raptor]

Make sure that renet properly detects the disconnect. Since you using a custom transport, it could be it.

You can check if replicon hadles it properly by checking RepliconClient::is_disconnected.

[dire aurora]

I just remove the RenetClient and transport entirely, so the transport used shouldn't make a difference at that point I think ... Afaict the run_if should trigger, but it doesn't ... My only guess is that the Local isn't local per schedule, and some other schedule "eats" the just_disconnected 🤔

[spring raptor]

Could you try this branch and run with enabled tracing?
https://github.com/projectharmonia/bevy_replicon/pull/234

[dire aurora]

Oh I think I found the cause ... I disconnect and connect within the same frame

For now I just forced a single frame between them ... Not ideal but it works I guess? 🤔

Being able to see the events and client stuff in trace is nice ... Tho those ack messages with the ticks look a bit confusing ... Are those bevy ticks? 🤔

[spring raptor]

Ah, bevy_renet::client_just_disconnected doesn't trigger in this case, right?

[dire aurora]

Yea, because it goes Connected -> Connecting

In the worst case you could even go Connected -> Connected, since my transport is kind of already connected the moment I insert it (I just don't set it to connected immediately)

[spring raptor]

It's RepliconTick. On client we don't apply events from server if client is behind on a critical change (insertion, removal or spawn). To avoid a situation when you receive an event, while the world archetype-wise is different.

[spring raptor]

In this case I would expect it to work, resource_added::<RenetClient> should trigger in this case.

[dire aurora]

Yea, resource_added does trigger, but it stays on connecting forever after that

Since the just_connected has the same issue 😂

[spring raptor]

Ah, I see!
We actually could fix it by additionally checking is_added. But it needs to be done inside renet.

[dire aurora]

Yea ... Tho replicon has similar systems that would have the same issue

[spring raptor]

Ah, right

[dire aurora]

It's kind of funny how in bundlication I never had these issues cause I just don't check connecting status 😂

Some of these bundlication impls I wrote really don't like the fact that I pass in Tick::default

[spring raptor]

Wait, no, we don't remove resources :)

[spring raptor]

What do you mean?

[dire aurora]

Replicon doesn't pass in RepliconTick to deserialize functions yet, so I "work around" that issue by passing in a default value ... Some of my functions were written with the (pretty reasonable) assumption that LastGround cannot be later than the current tick, or AnimationStart cannot be later than the current tick

Everything is later than default (0) so each of these functions was crashing

[spring raptor]

Ah, I see. I going to address it soon. But could you please check the API change of this PR first?
https://github.com/projectharmonia/bevy_replicon/pull/233
You don't have to review the code, just take a look at the changed examples in docs in replicaton_rules and command_markers.

[dire aurora]

After fixing those crashes it looks like a lot of things replicate fine ... Skills are broken tho ... Not sure if I forgot a replicated or if it's cause I had to change it the most of all my non-event replication

[spring raptor]

Awesome!

[dire aurora]

Also in case anyone thinks the "forgot a replicated" is a typo 😂

pub use bevy_replicon::prelude::Replication as Replicated;

[spring raptor]

Actually, no, the problem will be present since we reset on disconnect... I will open an issue to not forget for the future.

[spring raptor]

Yeah, it worth to rename. Maybe even with this release.

[dire aurora]

API looks functionally about the same, besides the extra structs when calling some register methods which is fine, at least now the types are checked better ... Would maybe be nice if the RemoveFn in CommandFns was also typed tho 🤔

[spring raptor]

Yep, just a better type checking to make write safe. Now you should be able to chain App calls nicely without unsafe blocks.
Structs a bit longer to type, but on the bright side deserialization in place now passed via builder pattern optionally. And shorter to construct defaults for manual impls.

I thought about adding a type to RemoveFn, but not sure how :(

At least the function is safe.

[dire aurora]

Worst case we could add a phantom data arg ... Wouldn't be very pretty tho 😂

[spring raptor]

Yeah 😅

Merging then.

[dire aurora]

Apperently I both forgot Replicated and my changes weren't correct (I didn't replace an old identifier map access with mapping entities) ... Besides the Tick thing I think everything works about as it did before the change ... Need to fix a few warnings and then it'll be time to make a rollback writer function 🤔
The changes: (ignore ~70 of the deletions, I spotted some unused code while switching over)

55 files changed, 995 insertions(+), 1104 deletions(-)

[spring raptor]

Opened a PR:
https://github.com/projectharmonia/bevy_replicon/pull/236

[spring raptor]

Awesome!

[dire aurora]

The changes are pretty reasonable compared to that +3k -4k nightmare when I did renet -> bundlication 🙃

[spring raptor]

That's a lot 😅

Two things left from my side: provide RepliconTick in serialize and add a history switch.
Going to work on it tomorrow, 3 AM for me now :)

[spring raptor]

@dire aurora opened another PR with a small change for markers API. No functional changes, just remove CommandFns::new in your code and pass functions directly:
https://github.com/projectharmonia/bevy_replicon/pull/237

[dire aurora]

Ah I forgot to respond, but these changes look good to me

[spring raptor]

Great!
Working on context argument with tick access, will ping when I'm done.

[vapid badge]

Is there a reason ClientId doesn't derive reflect? I have a couple places on the server side I use it as a key in resources and it would be great to be able to see those types in the inspector. I don't think I can easily derive reflect on those resources since ClientId doesn't

[echo lion]

Feel free to open a PR :)

Otherwise, one of us will get to it for the next release

[vapid badge]

https://github.com/projectharmonia/bevy_replicon/pull/238

[spring raptor]

@dire aurora Please, try this branch: https://github.com/projectharmonia/bevy_replicon/pull/239

[dire aurora]

The name for write/deserialize context name is a bit long, and the non_exhaustive makes it impossible to construct them for tests ... But maybe we just need a smaller scoped version of what test_app does specifically for testing the effects of the functions, without all the extra complexity of having two apps running 🤔

[viscid jacinth]

You're using a shared ComponentId to figure out how to serialize/deserialize a Component, right? But isn't it possible that the ComponentId won't be the same between client and server since they are not using the same world?

[dire aurora]

ComponentId doesn't actually decide serialization/deserialization, since there could be various way to serialize/deserialize a component with the new replication groups. The identifier used when communicating is a deterministic identifier that matches if both sides registered the same functions

But yea ComponentIds would almost certainly not match between a separate server and client app, and I wouldn't be surprised if they aren't even deterministic if they are both the same app

[dire aurora]

I haven't looked at replicon's exact logic, but in bundlication I used to just sort everything by type path and assign ids based on that sorted order

[spring raptor]

Correct, we use our own ID.

[spring raptor]

The name for write/deserialize context name is a bit long
Agree! But do you have other name suggestions? I really tried to come up with a nice name, but there are 3 contexts 😅

the non_exhaustive makes it impossible to construct them for tests
Ah, right... Let me think if we can keep it and somehow provide a convenient API for tests.

[spring raptor]

How about to have an extension trait for EntityWorldMut? Methods that will serialize C, write C, remove C and despawn entity. And hide access to fns from public API.

[dire aurora]

I just aliased it as both DeseiralizeCtx and WriteCtx 😅

[dire aurora]

If serialize just gives some binary data there, and write takes it I think that would be fine yea 🤔

[spring raptor]

Hm... Maybe just create an alias, yeah...

[dire aurora]

We always send entity updates as a single packet with all the unacked changes right? 🤔

[echo lion]

Spawns/despawns and component insertions/removals force sync points in the data stream, but component updates are potentially multi-packet/unsynced between those points.

[spring raptor]

And we split into several messages by packet size.

[dire aurora]

Hmmmm, so we have no way of actually knowing the latest server tick for which a component was up-to-date? 🤔

[spring raptor]

We know this tick per-entity

message_tick is basically a tick for which this message was sent. When we apply it, we remember this tick for an entity and ignore updates with older ticks (right now, no history switch).

[dire aurora]

That's whats stored in ServerEntityTicks right?

Or is that the opposite for the server's acks? 🤔

No wait it would need some client identifier if it was for the server

[spring raptor]

Yes

ServerEntityTick is a client resource for ticks from server for each entity.

[dire aurora]

So if I have a value from 22974 ticks ago, but the value in ServerEntityTicks says this entity was updated last tick, can I assume that old value is still the current server state?

[spring raptor]

Yes

It means that it didn't change for 22974 ticks.

[dire aurora]

Allright then I think I should be able to implement my rollback code correctly ... Since I need it to fall back to predictions if the rollback target has old data for that entity

Well except ofc that replicon won't pass in old updates, but that's not a super big issue while I test on localhost 😂

[spring raptor]

What do you mean?

[dire aurora]

If replicon receives old data it won't write it, so if I roll back to a tick where that happened the code would assume the old value is correct (which might not be the case) ... But if I'm testing locally without any ping variation, replicon will receive updates in-order and thus write every value

It's the missing history switch

[spring raptor]

Ah, sure, will tackle it right after the PR with contexts.

[dire aurora]

I should probably write some unit tests for this rollback stuff, my write function is over 40 lines, and the function to load predictions or authoritative data it over 50 😅

[spring raptor]

Yeah, tests are lifesavers 🙂

[viscid jacinth]

that only works if the component is registered in a plugin that is shared between client and server right?

[spring raptor]

Correct

[spring raptor]

@dire aurora I updated the PR, please try it: https://github.com/projectharmonia/bevy_replicon/pull/239
You will be interested in test_fns module (see the doc example) and you can also take a look at tests/fns.rs for how I use it for my integration tests.

[vapid badge]

To my understanding, the client side for replicon only has a notion of repliconticks that have been recieved from the server.

For folks that have done things like client prediction what are you using to track inputs and the updates that ack them. I could put another sequence number is my inputs and the replicated transforms that come back but it seems duplicative.

Based on the chat history I think there might be some plans for this, but I don't fully understand them

[waxen barn]

Yeah I think for prediction to be done correctly the server needs to respond back with the last input tick/id it processed for that client

[vapid badge]

And the client needs to be ticking independently, rather than only updating on server updates right?

[waxen barn]

thats correct, (to preface i dont use replicon). But what I do is this:

• create a PlayerInput object and set its values based on user input, and its seq id
• pass the PlayerInput into a function to process it (this will move your character, etc)
• put the PlayerInput into an array, and remove old inputs (I keep up to a seconds worth of player input history)
• send the PlayerInput over the wire
• when you receive the server's snapshot, it should have the acked seq id of the PlayerInput it last processed for you
• lets say it's a character you control, you set it's position to the snapshot's position, as well as the snapshot's velocity (trust me on the velocity lol)
• re-process all player inputs that come after the acked seq id

that will work pretty smoothly on its own but you'll want to decouple the visuals from the actual position so that you can smooth it a bit over time when dealing with larger corrections.

to be honest im not sure how you can achieve this without that acked input id

[dire aurora]

The client has its own independent clock yea, for client prediction it should ALWAYS be ahead of the server. If for whatever reason the server sends you data that's ahead of your client, accept its data, and increase the relative speed on the client so it's ahead again (by whatever margin you decide, but it should be at least half rtt + 2x the variation on that half rtt + 1 extra tick for packet loss), you can send data back from the server to help the client be aware with how far it really is ahead/behind so the client can do the opposite (lower relative speed) to be less further ahead when necessary (because being ahead increases latency)

[vapid badge]

I guess my question is why not have the client increment it's replicontick via the same policy as the server and rephrase the recieve pipeline to be based on the last acked replicon tick.

How it is now I'll be adding another logical clock with the same tick policy, no?

I guess the second half of your point counters that. It seems more similar to how unreal does its vaguely realtime adjustments, rather than a logical clock that's a bit closer to a rollback system

If I was to create a rollbackish system that expected a totally ordered simulation, I'd be recreating the replicontick tick policy on the client side

[dire aurora]

The clock situation is generally kind of awkward, can't really use replicon's tick because 1. It doesn't exist on the client; 2. It's not designed to go backwards or be changed without going up

But it's also not like there's a built in system for this, so you kinda need your own ... Which is really annoying with plugins that also need a plugin (like my rollback system I'm building that only depends on replicon)

[vapid badge]

I can work around not going backwards by always resimulating to the current client tick, but I'd need the tick to increment on the client independent of the server

[echo lion]

The replicon tick is not a clock, it only increments when you want to check for replication on the server, and is used to track acks and synchronize updates.

[vapid badge]

Does that not behave as a monotonically increasing logical clock?

That's all I need

[echo lion]

Only if you use TickPolicy::EveryFrame and every frame is the same duration.

Otherwise it's just an arbitrary selection of points from the server's timestream.

[vapid badge]

I thought you'd say that, I was planning to manually tick in fixedmain

I understand since that's not a part of the API you'd hesitate to provide it on the client side to avoid unintended bad behaviour

[dire aurora]

You can't resimulate just the current tick. If you get server data it will be somewhere in the past, you'd basically need to turn the clock back to have it behave correctly, since that tick value is your only timekeeping system, and not turning it back would desync both sides if you had a component that despawns after X ticks

[vapid badge]

No I need all the inputs since my last acked tick, resimulate all ticks since then with the updated position, and then keep going from my current tick

I don't need to actually increments the ticks along side those simulations since they have a fixed delta in my case

I'd phrase my simulation around inputs and the fixed delta, and the ticks / history is external to that

[dire aurora]

I mean it's simple enough to know from where you need to play the inputs, you got the message tick, every input before then should either have happened or been discarded, every update after that still needs to happen

[vapid badge]

My plan was to shove a custom schedule before fixed main that runs all the resimulation loop before the next fixed main (or does nothing)

[dire aurora]

Yea this is how it normally works: You get server state, roll back to the message tick of that server state, and resimulate what happened after that message tick again, only after that do you run your next fixed update (if it's ready anyway)

[vapid badge]

I think that assumes my RTT < my fixedupdate interval. Without a local tick at the same rate I can't disambiguate if I have multiple queued inputs which ones were actually acked by the update I just got.

Like if I'm 200ms ping, with a 10ms tick, I could have 20 inputs but the update is only acking 10.

Because I sent more inputs while the response took 100ms to get back to me

There is no ordering if the logical clock is only incremented by 1 party and I can queue inputs

I'm likely better off having my client create it's own sequence number for it's inputs, then the combination of that and the server tick create a valid vector clock

[dire aurora]

The client timestamps what tick their inputs are for

[vapid badge]

That's not enough if the tick implies a delta time. How would the server know how many ticks the 10 inputs with the same "last known server tick" are for.

That's relative to a varying ping

If I have a client side sequence number, and a know the clients tick policy, I can use that to get accurate delta times for the inputs, and back stop that by the elapsed server ticks since last input to avoid time cheating

[dire aurora]

You don't send the last known server tick, because that wouldn't actually tell the server anything

[vapid badge]

What tick do I send then? Replicon tick only ticks server side

I think we're saying the same thing

[dire aurora]

Like I said, you need a client tick, it just doesn't make sense to use the replicon tick for that, because it serves a different purpose from a clock

I run the same tick on both my client and server, and I update the replicon tick based on that

[vapid badge]

I originally came at this with the plan to tick replicon at the same interval as my simulation, in which case it would function as the vector clock I described.

I accept that requires extremely config, and I'd be better of making a client side tick specific for this purpose

[dire aurora]

This tick problem is wider than just bevy_replicon ... Ideally we'd have a tick value that's also accessible by other plugins that aren't necessarily replicon-specific

[vapid badge]

You could always go the unreal route and sync your clocks and use that. Bounded clock error would be good enough for many games in sure

Thanks for the chat :). Time to feed my son lunch

[dire aurora]

Most games do try to sync their clocks, but for client prediction it then still needs to be ahead ... I'll see if there's any support for some official Time<Simulation> or whatever that simply tracks the simulation's time 🤔

[vapid badge]

With a synced clock you don't do anything special, you're always RTT/2 "ahead" of when the server gets your inputs, so you can predict based on real time

[dire aurora]

You can't always be RTT/2 ahead, since ping isn't constant. Inputs also need to be for a specific server tick, since all alternative solutions create various problems, which means you need to be further ahead still if any packet loss is present

[spring raptor]

Technically, it exists on client, but I use it to track last received tick from server.
And it's kinda possible to make it go backwards - just insert the received value from server. This way client value (bigger) will be overwritten by server's (smaller).
What if I use something else on replicon side for tracking last received state, will it be possible to use replicon's tick on client too for third party crates?

[dire aurora]

Using replicon tick would only really work for replicon-specific plugins ... I'd imagine there's plenty of other usecases that need info about the simulation

Tho I do mess with replicon tick on my client (because it uses the same system as the server) so I think using a different resource to track that received state is probably a decent choice either way 🤔

[spring raptor]

So it runs on client too? 🤔

I mean increment

It could break your updates.

Replication will work, but updates may be weird.

[dire aurora]

Yea, I assumed the client didn't use it so it probably didn't need a feature flag 😂

[spring raptor]

Okay, I will change 😅

@dire aurora will it be okay If I use part of your message #networking message in the readme?
I have exactly the same thoughts about modularity and you expressed them so well. PR: https://github.com/projectharmonia/bevy_replicon/pull/240

[dire aurora]

Ah, yea that's fine with me

[spring raptor]

Okay, waiting for your feedback on the contexts PR. Let me know if it works for you.

@vapid badge About the tick, I think we can move tick https://docs.rs/bevy_replicon/0.24.1/bevy_replicon/server/enum.TickPolicy.html into RepliconCorePlugin and run default increment systems on both client and server.
And internally I will use a separate newtype (LastRepliconTick?) to track last received tick from server to avoid unexpected side-effects for users. And this should unlock the tick usage for you. You can't decrement the tick, but you can replace it with the one received from server and I think it should work for you, right?
If I understand correctly, @dire aurora working on a general-purpose plugin that he hooks into replicon, so it doesn't matter for him.

[dire aurora]

My code still uses bundlication's tick 😂

[vapid badge]

I would probably use ServerEntityTicks (which I assume you'd make based on LastReplicationTick ) and compare that to whatever the current RepliconTickis on the client side, while using the same tick policy on client and server.

[dire aurora]

ServerEntityTicks is just the last time an entity got an update

The LastRepliconTick is the last time an init update got applied I think ... Which contains spawn messages and stuff iirc

[vapid badge]

Which would be updated if the component I care about just got an update acking some of my inputs, seems like what I want.

I will likely start by implementing this with a sequence number of my own so I have something working to better validate how effective it might be to push all the logic down into replicon's tick system like this, but that's how I imagine I'd do it with the tick system.

[spring raptor]

Correct

[dire aurora]

ServerEntityTicks goes up without your component necessarily getting updated, but it does represent the most recent tick that that data from the server was valid

[spring raptor]

Yes

[dire aurora]

That also means you can't just apply updates from replicon to your components directly and assume you can savely resimulate from that, replicon might've updated only Transform and left the rest the same, while the client's state which is ahead could have other values there

This is where writers come in handy, if an entity gets updated you can load it from a history or a Remote<T> component and ensure it all matches the latest server state

If you only predict the client a history is probably unnecessary and just a Remote<T> would do

[dire aurora]

I get this funny error ... I'm not really sure what id it's talking about since I'm calling a method on an entity already 🤔
writing data into an entity shouldn't fail: Custom("The given id is not a valid entity.")

[vapid badge]

Ya I guess I need my inputs to be tagged with the clients local tick, but the updates also need to contain what client tick they were for, so really I'm just abusing the replicontick on the client side to avoid building my own fixed interval clock. I.e the client tick and server tick don't have a causal relationship (though they can form a vector clock).

I'll be better off with my own logical clock.

I'm planning on having my Transforms replicate to a component other than Transform so I can control how displaying state from the server and state from the local simulation works. I'll have to look into what you're referring to more

[dire aurora]

For the thing I'm referring to it's basically like this: If Transform didn't change since your last ack, replicon won't send it. But that entity might still have updated so ServerEntityTicks for that entity will increment

[vapid badge]

Yes totally understood, I was referring to the "writers"

[dire aurora]

Luckily main has write functions to write them elsewhere so you can always find the latest server value for each component

[spring raptor]

Yes, just don't use ServerEntityTicks. It's last updated tick for an entity. I pass the correct tick for each component into writing function.

[dire aurora]

Ah yea, that's the thing on main. You can register a marker, which when present will make replicon check if your component has a different function to write the value. That function can then just write to Remote<T> or any other type instead of directly to T

[vapid badge]

That sounds extremely similar to what I was thinking.

Thanks for all the feedback here and in #networking . I'll see how much of this I can get through when I have childcare again early next week 🙂

I think at this point I need to actually implement more of it and see where I end up

[spring raptor]

This is strange... The error comes from Bevy and it basically means that the entity is not valid. How you even triggered it? :) I have a bunch of tests + the example that works...

Do you despawn your entity somehow inside your writing function?

[dire aurora]

I just do this 🤔

let mut entity = app.world.spawn_empty();
entity.apply_write(&[1, 2, 3], components[0], tick);

The function I pass in takes 3 u8s, which is why it's 1, 2, 3

[spring raptor]

Could you send me your test function?

Does it work the default functions?

[dire aurora]

Oh, apparantly I forgot to insert my replication fns ...

But why does that give me an error about an entity?

    let mut app = App::new();
    app.add_plugins(bevy_replicon::RepliconPlugins);

    let mut replication_fns = ReplicationFns::default();

    let rule = BundleWithAttributes::register(&mut app.world, &mut replication_fns);
    app.insert_resource(replication_fns); // This line fixed it

    assert_eq!(17, rule.priority);
    let components = rule.components;

    let mut entity = app.world.spawn_empty();

    let mut tick = RepliconTick::default();

    entity.apply_write(&[1, 2, 3], components[0], tick);
    assert_eq!(
        entity.get::<Transform>(),
        Some(&Transform::from_xyz(1., 2., 3.))
    );

[spring raptor]

Hm... Maybe it's UB?

But I have checks inside ReplicationFns::get.

It should fail here: https://github.com/projectharmonia/bevy_replicon/blob/ad083ba3240acfa4f010c612caf2c2c93e1b0a76/src/core/replication_fns/test_fns.rs?plain=1#L137

Wrong link

Fixed^

[dire aurora]

We don't actually check that it's from the same instance ... Do we register any default ReplicationFns? Maybe for Entity?

[spring raptor]

Nope, it's empty by default.

We don't check if it's from the same instance, yes, but I assume that you should have out of bounds in your case...

[dire aurora]

Yea that's what I'd expect too, but somehow it finds a function, and calls it, and presumably this data just makes it complain about an entity 🤔

[spring raptor]

🤔 I will debug

[spring raptor]

Defenitely a memory issue, I having

|| assertion `left == right` failed: pointer is not aligned. Address 0x1 does not have alignment 8 for type bevy_replicon::parent_sync::ParentSync
||  1
||  right: 0

Ah, right, we have ParentSync.

We register it as part of ParentSyncPlugin.

So it have a registered component by default and it tries to use the wrong function if user uses a value from a wrong instance

[dire aurora]

Well not too big of an issue then, it's not actually unsafe it just reads your data as the wrong component

[spring raptor]

@dire aurora everything else is good?

If yes, I will mark it as ready for review for koe

[dire aurora]

Bevy_bundlication finally has working tests now at least ... If there's other issues we can always fix them later

I basically just need to add tests for rollback at this point, and then I have everything working (besides some minor details like the history switch, the slightly less optimized formats due to lack of component visibility layers, and I need to make a PR with some feature flags for some minor compile optimizations)

[spring raptor]

Great, marked as ready for review!
@echo lion it looks big, but it mostly just turning some write arguments into a context and passing it around. And a context with ticks for other callbacks. Most of "+" lines are just integration tests.

[spring raptor]

For inputs you include client's tick and server sends you last received value back.
For interpolation/rollback I would imagine that you write your component C into something like Remote<C> / History<C> and store the received C with the its server tick.

So I think for this use case you can use replicon tick on client...

[rotund vine]

can replicon synchronize resources?

[dire aurora]

I don't think so, but you could probably send the resource value via events or make it an entity (and query it with .get_single()) 🤔

[spring raptor]

Correct, right now we don't have this feature. It's easy to add, I just never needed it 😅
Feel free to open an issue, I will take a look someday. We just need to add something like replicate_resource and store their IDs inside ReplcationRules.
Then add one more iteration here: https://github.com/projectharmonia/bevy_replicon/blob/a06c45b0eacf9fede41e225701d61b026c5e7eeb/src/server.rs#L232
And then logic is similar to components, but simpler, you just iterate over resources from https://docs.rs/bevy/latest/bevy/ecs/storage/struct.Storages.html
Our replication message is dynamic, so you will write one optional array that won't be send if it will be empty.
And similar logic on client.
For removals you will need to track removals somewhere like we do with components. Maybe some separate tracker like we do with components that we iterate and write removed IDs.

[iron flare]

So, renet doesn't have a thread so I'll just ask here. is it possible to have a client connected to more than one server at a time? and if not, is there another messaging backend I could use that does support it?

[spring raptor]

Renet have #1038137656107864084, but I can answer it here - no, it's not possible :(
What is your use case?

[iron flare]

I want to try to make a mesh network plugin using replicon, which will basically have multiple server "nodes" and a master server. each node will have a position and radius which determines the area of the world where it holds authority (replicates entities etc). I want these areas to overlap slighty, and in those areas you are connected to more than one server. this would allow for the client to not have to wait for the switching of servers

[dire aurora]

I'm not sure if an overlap is the way to go, but you'd definitely need multiple connections at a border anyway ... The fact that you'd have two sources of server entity ids might also be an issue 🤔

[iron flare]

yeah it will be challenging

but if I figure it out, it could be an awesome plugin.

[spring raptor]

In replicon you read messages from RepliconClient. You can feed it from multiple sources...
In theory you can use something like bevy_renet and bevy_quinnet at the same time.

But I think for your case you need something that supports P2P well.

Maybe try to provide a replicon integration with something like https://github.com/johanhelsing/matchbox ?
It will definitely require some changes to replicon, but I will be open to it

[iron flare]

I got started on doing a implementation for replicon w ith quinnet, but it doesnt tell you what channel the received message is from, which is needed by replicon

[spring raptor]

That's a problem... It's important to know from which channels messages comes from to treat its bytes correctly.
Maybe ask authors to provide it receive_payload_on method similar to their send_payload_on?

Does it support multiple connections at the same time, btw?

[iron flare]

yeah it supports multiple connections, thats why I wanted to use it. I'll see if I can fork it and open a PR

[spring raptor]

Feel free to ping me if you need something from replicon side.
Also I remember that @covert fog was interesting in something like this.

[iron flare]

alright, if I manage to create a quinnet replicon layer, should I open a PR at the replicon repo for it?

[spring raptor]

I would prefer to maintain only a single layer that I use and let other people maintain what they use.
But feel free to open a PR to add your repo to the readme here: https://github.com/projectharmonia/bevy_replicon?tab=readme-ov-file#related-crates

[dire aurora]

Hmmm, we changed ClientMapper to commands, but it can be a bit annoying to not have access to it (I have some things I have to map later since they're stored as Vec<u8> but might contain entities) ... I wonder if it makes sense to have a ClientWorldMapper and maybe even a command to map a given T after it was spawned 🤔

[spring raptor]

It's functionality now a part of the context, you can call map_entities on it.

[dire aurora]

Problem is the context only exists when deserializing/writing

I need to map with ServerEntityMap much later

[spring raptor]

Ah, I see... Okay, let me bring it back.

@dire aurora What behavior do you need when there is no such entity?

[dire aurora]

If I'm mapping with &mut World they could be spawned like usual I guess, same if it were a command ... I don't think map_entities even really allows fallible conservsions 🤔

[spring raptor]

You may expect it to be exist since it's done after replication?

But could you explain the use case a bit more? Why do you need to map a component after spawn? 🤔

[dire aurora]

It most likely should exist, but there's a small chance it doesn't, and it could still be handled like usual

My usecase is pretty much: I have status effect entities, but I don't replicate them, instead they are added to a map on the entity that is affected by the status effects, and the status effect's values are stored in there. Some of these status effects have an entity in them, so it ends up on the client with the server's entity still there ... It might sound a bit weird that they'd have entities in them, but it's cause status effects can be and do anything, it could even be a skill you cast (and skills are entities) or an effect that links you to another player

I guess in general there's a decent chance of it happening when generalized systems are involved ... I've had it on older systems too but those have either been simplified, or folded into status effects

[spring raptor]

But how do you send them? In serialization function you have access only to the component Ptr.

[dire aurora]

They are a Vec<u8> on both sides

I then later call a function to deserialize that into my real component

[spring raptor]

So you server you create a component with Vec<u8> and you replicate it instead?

[dire aurora]

Yea, because I don't replicate the entities that hold the components, so I can't actually replicate the real value directly

It's all typed erased

[spring raptor]

🤔 your goal is to avoid sending Entity to save bandwith?

[dire aurora]

Yea, and also to avoid having to sync up all of the entities

But some of these values still refer to a server entity, so I need to be able to map them

[spring raptor]

Maybe we need something built-in for this in the future...
I use entities for tasks in my game, but don't optimize it like this 😅

I will add command and world mapper in about 2 hours. In the meanwhile just copy the ClientMapper code into your repo :)

Mappers could be useful for other things, worth to have.

[dire aurora]

Yea, having the mappers around in a usable form just generally seems useful

And yea we'll probably need nicer abstractions for this stuff 🤔

It's a complex issue to solve tho, since things like rollback also tie into this

With my current system I don't need to rollback status entities at all, I just detatch (undo their effects) and disable them before I roll back, then only enable the ones that get attached again ... Still need to implement the logic to remove the ones that never got re-enabled after the rollback is done tho

[spring raptor]

@dire aurora added mappers

After the review from koe I will address history and tick tracking on client. Just don't want to have a huge PR or a lot of conflicts.

[spring raptor]

Pushed one more commit with new method to make creation more convenient.

[spring raptor]

Merged 🎉

[grave yarrow]

@spring raptor what are some features we'd want from lightyear that we don't currently have?

resource replication?

replication priority too is another bigger one I can see

[spring raptor]

Yes, resource replication and priority.
First one is quite easy, just rarely needed, so I didn't bother.
But second one is important to have. Other engines, like Unreal have this feature. But not something critical for development phase, so I left it for later.

For the upcoming release our focus was on providing a better API for prediction/rollback crates and replication cusomization.

For example, if you want to replicate Transform only if Player is present, you will be able to replicate_group::<(Transform, Player)>().

And you will be able to specialize serialization for groups.

Like "replicate only position field" for (Transform, StaticObject).

And do a full replication of Transform for other cases.

[grave yarrow]

what's the "better API for prediction/rollback crates" part about?

I was looking through the code for a bit and was feeling like it might be nicer to have more full control over the replication message if I was doing a prediction/rollback

[spring raptor]

In the previous release users could override serialization and deserialization logic. Deserialization function also had writing logic into an entity.
Interpolation and rollback crates like https://github.com/Bendzae/bevy_replicon_snap or https://github.com/RJ/bevy_timewarp relied on these functions.
But it wasn't very ergonomic. And if user want both, prediction/interpolation and customize serialization/deserilization, it would require to copy a third-party crate into custom functions.
Another pain point is that prediction and interpolation is a per client thing: you most likely predict your local player entity and interpolate others. So it would require using if else inside writing function that is not ergonomic and slow.

But in this release thanks to @dire aurora we developed quite a nice API.
First, we separated serialization/deserialization and writing. This way users customize their logic independently.
Second, we provided a marker-based API for quick check (and zero-cost if you don't have prediction or rollback) which writing function to use.
This way rollback crates just register a writing function for Predicted or Interpolated markers and it just works. Same users just register components as usual.

[spring raptor]

Maybe, but for prediction/rollback you don't need it.
The idea is to register markers like Predicted / Interpolated and provide writing functions. These functions will write component C into something like Remote<C>/History<C> and you will be able to interpolate/replay inside your crate or game systems.

@dire aurora is already working on a rollback and input buffering crates, so you might just want to wait a little.

[grave yarrow]

gotcha

[spring raptor]

I going to draft a new release soon, just need to address a few things.

[dire aurora]

Input buffering I'll probably be able to make a usable crate for soon, since my next step is to finally fix the bits of weirdness there as well as support things like receiving inputs for other players (can help a lot with prediction, and ofc deterministic replication), it should hopefully be able to support pretty much every normal usecase in games, unless low latency and avoiding dropped inputs are more important than preventing cheating
But rollback wouldn't really be in a decent state until we get either of these PRs merged: https://github.com/bevyengine/bevy/pull/12928 https://github.com/bevyengine/bevy/pull/13120 ... Since the former git got hit with S-Needs-RFC that's definitely not going to happen anytime soon tho 😅

This rollback approach is shaping up pretty nicely tho, still need to add a few more features to make it more correct, but it has been behaving a lot better than my old approach when I used bundlication with Remote<T> and comparing values before rolling back

[spring raptor]

Cool!

Second one looks a bit nicer in my opinion. Convenient feature, but why it's needed for rollback? Just curious.

[dire aurora]

When working with rollback you roll a lot of components with entity references back, if those entities don't exist everything will be completely broken. But you don't really want to keep entities around that interact with things just to avoid breaking things when you roll back. So the solution is: You don't despawn things, you disable them until they fall out of history; And in the other direction, when you roll back and some things don't exist yet, you disable them until they become active, this time despawning them if they didn't get re-enabled when you're "back to the present"

[spring raptor]

Ah, I see... You probably need to mention the use case, it's quite important.

I wondering if you can force-override an entity archetype into ArchetypeId::EMPTY, this way it will be untouched. Hacky, but could work.

[dire aurora]

And here I thought using a custom bevy branch was awful 😂

[spring raptor]

😅

[dire aurora]

Also the usecases are mentioned on the first pr, but the second doesn't really do anything by itself so the usecases are basically limitless

[spring raptor]

Sure, it's quite useful, but having an example use case could help.

Under "important" I meant that your use case is important :)

[dire aurora]

We even have a whole RFC listing usecases ... Tho making a good RFC is kinda hard since we haven't seen people use a feature like this in the ecosystem before (because it's not possible without cursed unsafe stuff like messing with archetype ids) https://github.com/NiseVoid/rfcs/blob/disabled_entities/rfcs/81-disabled-entities.md ... There's a decnet chance a single Disabled component doesn't actually make sense ... Especially if weirder architectures are involved like lightyer's multiple entities thing

[spring raptor]

Ah, I didn't see it.
The RFC is well-written. After reading it, I started to think that maybe Disabled is nicer then the second one.

[dire aurora]

It's nicer but it also has it's issues ... The second one is kind of a basis for Disabled and others, but having a single Disabled marker might not always be ideal

My rollback example from before might for example be better off if both of the reasons things can be disabled had a different marker

[spring raptor]

@dire aurora switched to a separate resource for storing last received init tick on client to avoid confusion: https://github.com/projectharmonia/bevy_replicon/pull/244
If you use any server event with a custom receive system, it will be a small breaking change for you (since you track master).

[dire aurora]

I use the default receive systems luckily

I'm actually surprised we even have the ability to change those, in bundlication sending events was a command, so no systems needed and receiving them all worked with one system (the same one handling receiving entities) ... I guess it's kind of similar to this optimization james did: https://github.com/bevyengine/bevy/pull/12936

[spring raptor]

Hm... Interesting. My use case is that sometimes I send reflected types and needed access to the type registry.
I guess we can do a similar trick with functions and map entities / deserialize reflect inside them.
I will open an issue to not forget, maybe for the next release.

[dire aurora]

Yea it's nothing too major, it could just be simplified and optimized ... Luckily bevy_replicon usually deals with only a handful of events, rather than the hunderds registered in a bevy app

[spring raptor]

@dire aurora last thing left to address is the history switch.
When we serialize an entity, we serialize its ID and the sum of the size of components (instead of the number of components). This way we can skip the entity by advancing the cursor.
When the history is enabled we can simply ignore this check and always call writing functions. But how to skip deserialization for components that aren't predicted or interpolated? We don't know the size in advance. Add comparsion logic into the writing function and just ignore the deserialized value? Not quite elegant... Maybe you have a better idea?

[dire aurora]

We can just call deserialize and discard the value right? Tho in bundlication I just generated "consume" functions, the difference there being that I didn't spawn entities when mapping failed

[spring raptor]

Yeah, probably worth to add a consuming function...
Since users register SerdeFns struct, we can make it defined by default for mapped and non-mapped constructors, so users won't need to deal with it if not needed, but can optionally override it via builder.

What I dislike though is that it will make writing more complex. But I guess it's unavoidable.

Also it's additional branch even when an entity doesn't need history...

[dire aurora]

The skip could be done on the replicon side already, before calling the write function

If the function it picks doesn't request history, it can just be discarded

I think we can also early out the entire entity if no markers that need history exist

[spring raptor]

Ah, yes, makes sense!
This will be much more user friendly.

One more thing about which I'm not sure. Right now each message includes relevant server tick. And we assume this tick for each entity in the message.
So an entity could actually have a component that was updated on tick A, but client can receive it as A + 3. Is this fine for rollback and interpolation? 🤔

[dire aurora]

Hmmm, it might be an issue for interpolation but one that could be worked around (by requesting history), for rollback it's fine since you don't care from when the data actually was, just what was correct at that point in time

[spring raptor]

Got it, thanks!
In theory you could miss an update for tick A and receive an update only for A + 3, so even history may be not very accurate. But I guess there is nothing to do about it.

[dire aurora]

Yea that's unavoidable, and the impact likely wouldn't be too large, it would just interpolate as if A-1 - A+3 had a linear change, rather than A-1 - A having a change, then A - A+3 being the same

Also very unlikely for it to be 3 ticks off with history 😂

[spring raptor]

Thanks, I will start working on it, will let you know when I have an early working draft for feedback on the API.

[covert fog]

quick question, what is the difference between interpolation and rollback?

[dire aurora]

You can't even really compare them ... Interpolation interpolates between received values (and usually also buffers/delays them to some extent) with the goal of avoiding jitter ... Rollback is a huge ball of complexity necessary when you want client prediction that's as good as possible, you turn back the whole simulation to when data was received, then replay it from there, conceptually not that hard implementation wise kind of a nightmare 😂

Plenty of games have both, in many shooters there's some simple interpolation on things the server sends (usually very minimal to avoid extra latency which cheaters could bypass), while doing full client prediction on only the player

There's also extrapolation, where you just go "they were at A previous frame, at B now, so in 2 frames they should be at D" without actually simulating what really would've happened

Basically no one likes extrapolation tho, because it's essentially just a compromise and can easily introduce large noticable artifacts

[covert fog]

Okay, so rollback is related to extrapolation. Both involve driving the state forward on the client. The difference being that with simple interpolation, when new data comes from the server you just say "okay this is the state now," but with rollback you take into account the network delay (requiring a buffer of past states to rollback to and fast forward)

and interpolation is just spreading out the server update over time, for visual smoothness.

[covert fog]

like rubber banding, if I understand correctly

or, is client side prediction mean something different than extrapolation

[dire aurora]

Extrapolation is the cheap low quality alternative to prediction + rollback. Sometimes it makes sense for games that can't optimize things well, have way too many things going on, but still need the player and the world to be on the same timeline

[covert fog]

Rollback sounds like overkill for alot of games.

But you kinda get extrapolation for free, just run physics on the client and replicate velocity

[dire aurora]

Yea, it's common to do it for the player, which you can do relatively easily because it avoids most of the edgecases and performance issues, but only games requiring very high levels of precision and fairness benefit from full world rollback

[covert fog]

I'd imagine you have to have either rollback or interpolation for the player. Otherwise there would be either stutter (if you use prediction) or delay (if you don't).

[dire aurora]

If you have client prediction you're gonna want to do rollback on the player, but there's also plenty of games that can just get away with only having interpolation and just waiting for the server (especially if it's unlikely people would play with people from other regions)

[grave yarrow]

rollback also keeps your game world's more consistent

[echo lion]

@spring raptor ready for a release? I'm planning to update bevy_replicon_repair/attributes and thinking it'll be easiest to skip past v0.24.

[spring raptor]

Almost :) I want to address history access before it. I will open a PR today after work.

[dire aurora]

I've tested most of the new features already too, they all seem to be working fine, tho I haven't tried introducing variable latency or packet loss yet (because without history that likely wouldn't end well)

[spring raptor]

@echo lion I have a working draft, but I need you to review this one first: https://github.com/projectharmonia/bevy_replicon/pull/246

Thanks!

[spring raptor]

@dire aurora I opened a PR and described the API.
Still need tow write docs and tests, but I would like your opinion about the interface since you will need to use it.

https://github.com/projectharmonia/bevy_replicon/pull/247

[dire aurora]

The API description sounds goods. But the consume_or_write implementation looks wrong, it could pick a lower priority writer if the one that should be used has no history

[spring raptor]

Ah, you are right, thanks!
I will proceed with tests and docs then.

[spring raptor]

Fixed and decided to pass a struct into register_marker_with instead of usize and bool, will be less confusing.

[spring raptor]

Added docs, will write tests tomorrow, going to sleep now. But I think the branch should be work, you can try to experiment with it now.
Will also need to update the history example in the docs to use this feature.

[grave yarrow]

do you have bundlication up anywhere with the replicon changes?

I see the repo but last commit was months ago

so I'm assuming it's mostly a local branch

[dire aurora]

I don't but I can push the changes

I pushed them

https://github.com/NiseVoid/bevy_bundlication/commit/2d7b69133d90457b3991c0b2ae72a5f4c4990a2d
A beautiful "remove almost everything" commit 😂

[dire aurora]

Oh it didn't even work correctly on bevy_replicon main ... Simple fix tho

[spring raptor]

@dire aurora done!
Would you like to test your rollback crate before the release first?

Would you like to open a PR to add your crates to https://github.com/projectharmonia/bevy_replicon?tab=readme-ov-file#related-crates ?
If you do it before the release, they will be displayed on crates.io.

[dire aurora]

Of course I immediately run into bugs because I never tested writing values in reverse order ... I should really add history write tests 😅

[spring raptor]

Yeah, for things like this it's quite hard to verify if it works correctly without writing tests.

[dire aurora]

Hmmm, I should probably make it so that the rollback actually rolls back to the frame of the oldest received data, when my latency is high enough it ends up always reloading prediction, which means it's perpetually wrong 😂

[dire aurora]

I think the history is being written correctly but I've realized a fairly major flaw with my approach ...

I'm loading history assuming the previous value is correct if the latest data has been later, but that's not necessarily true ... I can only assume that if I actually had data for that entity on that tick ... Should be relatively easy to add tho 🤔

[spring raptor]

Could you elaborate?

[dire aurora]

If I've received data for tick 5, and I try to roll back to tick 3, but have no data for tick 3, I currently load tick 2

But it's entirely possible that tick 3 was lost or hasn't arrived yet

That assumption is only save to make if the entity got updated on that tick at all

Actually I say easy but this might actually need some support from the replicon side for entities that have very few predicted components ... We'd basically just need to know which ticks were received for each entity that frame

[spring raptor]

Hm... That's true. So you need some sort of acks that a component didn't change from me?

I.e. you need to distinguish between "didn't receive" or it "didn't change"?

[dire aurora]

Yea, it's not really an "ack" but yea that's the crucial difference here

If I assume didn't change, but it instead wasn't received things become jittery

Knowing the received ticks would also help my rollback code decide to which tick to roll back (by just picking the oldest one for all predicted entities, then clamping to the last tick in history)

[spring raptor]

Got it. Need to think how to implement it nicely...

[spring raptor]

@viscid jacinth How do you handle this? ^
When you do rollback, how you distinguish between values for which you didn't receive an update and values that didn't change?

[dire aurora]

Working on correct rollback really makes me miss the simplicity of the old approach I had, it made for a very buggy experience, but it only needed to consider the latest server data so that was super simple 😂

[viscid jacinth]

I'm not sure I fully understand the problem, but I only check rollback for entities where I know that I received a server update (and I have access to the server tick for that entity update)

Basically, when I receive an update I write the updated server tick on the Confirmed component, and I only check rollback for entities that have Changed<Confirmed>

[spring raptor]

Thanks!

@dire aurora if I understand you correctly, that's the mentioned approach you tried before. What issues did you have with it?
About the mentioned example above, is tick 2 is the latest known value for this component from server?

[dire aurora]

If it's 2 this wouldn't happen since it would see 3 has no data and load the prediction

In this case it woud be 4 or 5

Technically 4 or 5 could be missing, as long as the server tick is >= 3, but then the data would be the same and loading 2 would be correct

[spring raptor]

Not sure if I understand you. Could you explain how the rollback mechanism works in your crate?

[dire aurora]

When a rollback triggers (currently it's hardcoded to always do so, going back 5 ticks) it checks the authoritative and predicted history for every value, if the authoritative history is valid (currently decided based on if the server tick is after the tick that is being rolled back to) it will load the latest value in history. If the server tick is 4, the tick that is rolled back to is 3, and the last data is from tick 2, it will load tick 2's data. If it decides there is no valid data it will instead load the predicted value

This is a bit oversimplied, technically the value from the tick before is loaded, since data from tick 2 is from after the update. But accounting for that makes all the numbers confusing 😂

The decision in that case to load tick 2's value however would not be correct, since the data for tick 3 never arrived, and tick 4 had different data (if that component was excluded on tick 4 it would mean 2 is still valid I think)

[spring raptor]

So you assume that tick 3 have the same value as 2, but it's not always the case. But why rollback to 3, maybe just rollback to 2?

[dire aurora]

If I just rolled back the player rolling back to 2 would work, that's basically how my old approach worked (actually it would've gone to 4 instead, that was the latest received data here) ... But that's not possible when rolling back a lot of entities

We might've received tick 4 for the player, tick 3 for a party member, so we need to go to the oldest received tick, and end up at 3

[spring raptor]

Also not quite understand why you rollback before the received tick (you mentioned that you received tick 4, but you try to rollback to 3)...

Looks like lightyear rollbacks just to the latest received value and just replay all the inputs until the current client tick.

[dire aurora]

rollback before the received tick
That's just the roll back only the player vs anything difference

[spring raptor]

Ahh

So for everything except player you rollback to the latest received tick. But for the player you doing it differently?

[dire aurora]

Going back to the oldest received data is also necessary to ensure anything that isn't networked is in a valid state. For example you started a skill, but canceled it on tick 45, received data for 46 and 50. If you load 50 it might look like the skill canceled correctly, but the server missed that input and the skill already ended by itself. So now the cooldown is wrong

[spring raptor]

Ah, I see

[spring raptor]

But you cancelled it on 47, how loading data for tick 46 helps?..

[dire aurora]

I mixed up the numbers, fixed it 😅

[spring raptor]

Got it, so you will apply 46 and delay skill cancellation by one frame in this example?

[dire aurora]

Yea and that might make it so that it doesn't get canceled, but it's pretty fast so it already finished. So the server says your skill is on cooldown, but the client says it's not

[spring raptor]

Okay, I understand why rolling back to the oldest received value is important!
But why do you need to go beyond that sometimes? In the example above when you said that you received tick 4, why are you trying to rollback to 3?

[dire aurora]

Rolling back to 3 there is necessary because the entire world needs to roll back to the same tick, if the oldest tick for one entity is 3 while another is 4, it needs everything to go back to 3

And then there's the common way of debugging rollback that just forces rollbacks too (which is how my game runs atm)

[spring raptor]

Ah, I see, you mentioned it before. So you rollback the entire world...

Now I think I understand :)

Okay, what you need to do is to have a way to distinguish between "no change" and "missed change"...

@waxen barn I have a small question for you now :) Do you rollback each entity separately or the whole world?

[dire aurora]

Yep, and no change vs missed change is not something we can always know for sure, but if we have data for that entity for that specific tick, and it didn't have this component, we know the last value was valid (otherwise we would've put it in the update again). And I think that can also be extended to "future" ticks that got received with no data for the component

But if the tick that is being rolled back to received no data, and there is either no future tick or it had different data, then we have no clue if the last value was correct, and are better off using the prediction

[spring raptor]

Yes, this is absolutely correct.
But I understand correctly, it's not enough, you have jittering?

[dire aurora]

Right now my code has no clue about which ticks got received for the entity, only the latest received tick

So it will assume all old values are correct, which means every time it rolls back to a gap jitter happens (unless things were actually stationary the whole time)

I could try to track that in write functions (by queueing a command that sets that tick as received), but that presents the risk that it almost never rolls back correctly on entities with very few predicted components

[spring raptor]

We provide ServerEntityTicks, will it work for you?

It's a map

[spring raptor]

Could you explain this part?

[dire aurora]

That's where I get the latest received tick from, but since multiple ticks (especially with some variation in ping) can be received in one frame I can't use it to figure out all received ticks

[spring raptor]

Ahh, you need all received ticks, I see

[dire aurora]

If my write function adds a custom command that just sets that tick to received it would be set only if at least 1 write function for predicted values gets called, but there might be some object that only has one or two predicted components

[spring raptor]

Got it now.
So you think if you somehow get access to all received ticks for an entity, it would solve the problem?

I think it should be nicer then send updates from server each time...
But what if an entity just didn't change or you have only a networked component and it's predicted?..

[dire aurora]

If the server never sends updates it would probably break yea, I would need some very weird non-deterministic stuff to happen for that to actually cause problems in practice tho

Simpler entities would probably only updated as side effects of other things after all

[spring raptor]

Got it! Sounds like a good compromise compared to sending "didn't change" from server. Also should be easy to implement.
I will think about the API, but if you have an idea how user interface to access all ticks instead of the latest one should look - feel free to suggest.

Maybe event or maybe store serveral last ticks inside ServerEntityMap

Or maybe based on marker (you need it only for predicted entities, right?)

[dire aurora]

Yea, I only need the data for predicted entities ... I don't really need to worry about which tick the floor was last updated (tho the server does send that)

[spring raptor]

I probably should unite it with the added history parameter. If history is requested, provide user with each received tick for an entity.
How to provide it nicely for users is also a question...

[spring raptor]

I have one more question about it for better understanding. You previously mentioned that you have interpolated entities. For these entities you just use the interpolated value when you rollback?

I added TickConfirmed event as a quick workaround for you to test if using it fixes the jitter. Could you check it?
https://github.com/projectharmonia/bevy_replicon/pull/247/commits/aec526043c2fd4857dc5b8cabd2507c5033a6fb4

[dire aurora]

I don't have interpolated entities atm, but anything that doesn't get predicted would probably need to be interpolated (especially common when you only roll some things back). If I did it would probably just load the best value it can get, then letting interpolation set it to the correct value again later (since it needs to update every frame anyway)

[spring raptor]

Got it, thanks

[spring raptor]

@dire aurora I found that in this discussion they mention that without interpolation for predicted entities it will jitter: https://discord.com/channels/691052431525675048/1207166628811505694
Maybe the lack of interpolation causing this?

[dire aurora]

I think this actually involves interpolation on top. That's not something I currently do, tho eventually I might need to since the 60Hz fixed update can cause some of its own jitter

Interpolation and smooth error correction are both still things I need to look in to at some point, but the rollback should at the very least be able to give the same result as running 60Hz locally as long as no mispredictions happened

[spring raptor]

Makes sense, but I see that periwink even tried 256 hz and still don't have smooth experience because FixedUpdate not always does the same number of steps.

[dire aurora]

Yea, FixedUpdate has some jitter by design, I've tried to work around it in the past but it's a pretty complex problem to solve when you want something that works on any tickrate with any refresh rate

I do most of my testing at 60Hz with a 60Hz tickrate, which at least makes it so that when I'm lucky I can't see the problem 😂

[spring raptor]

Got it, so it should work...
Okay, can you try if the access to all ticks via the mentioned event solves your problem?

[dire aurora]

Working on it, writing the actual logic is kind of a pain tho 😂

[dire aurora]

Hmmm, I'm managing to make it avoid jitter sometimes now but not always ... I guess I finally need to clean some of the hacks in my code up and write some good unit tests that cover all cases instead of only 3 😅

[viscid jacinth]

Btw, the interpolation on top of prediction is not actually that important to have (at least in an initial implementation). You usually onlynotice the jitters if your camera follows the predicted entity (i.e. camera.transform = predicted.transform), but otherwise it looks smooth

[spring raptor]

Okay, I will hold the PR and the release.
Looks like Periwink uses only latest tick, curious if having access to intermediate ticks to confirm that there wasn't any changes helps.

[dire aurora]

The solution I made to store the confirmed ticks might actually make sense for how replicon stores it ... I store the most recent confirmed tick, and a bitmask with the 64 ticks before that

The code is fairly short:

pub struct Confirmed {
    mask: u64,
    last_tick: Tick,
}

impl std::fmt::Debug for Confirmed {
    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
        write!(f, "Confirmed [{:?} {:b}]", self.last_tick, self.mask)
    }
}

impl Confirmed {
    pub fn get(&self, tick: Tick) -> bool {
        if tick > self.last_tick {
            return false;
        }
        let ago = self.last_tick - tick;
        ago > usize::BITS || self.mask & 1 << ago == 1 << ago
    }

    pub fn enable(&mut self, tick: Tick) {
        if tick > self.last_tick {
            self.resize_to(tick);
        }
        let ago = self.last_tick - tick;
        self.mask |= 1 << ago;
    }

    fn resize_to(&mut self, tick: Tick) {
        let diff = tick - self.last_tick;
        self.mask <<= diff;
        self.last_tick = tick;
    }
}

[spring raptor]

You suggesting to store last received tick for entities as a component?

With previous ticks

I.e. like this 😅

[dire aurora]

That's how I do it now, but it could also be placed in the map we have now

[spring raptor]

Maybe having it as a component will be more convenient...

[dire aurora]

I guess both have different tradeoffs, with a component you don't have to look it up in the map, but then you have to insert a component, and we don't have good batching so that's a performance hit on every spawn

I guess we should be able to always spawn it together with Replicated on the client side tho 🤔

[spring raptor]

Right. Then I will do it like this.
Thanks for sharing!

I think it will be nicer to do it separately from the history branch, the change is unrelated...

Reverted the event hack. @echo lion could you review https://github.com/projectharmonia/bevy_replicon/pull/247 ?
I will open another one after it.

[echo lion]

Ok

[spring raptor]

One more thing I realized just now. I emitted events only for updates. I should have fire them also for insertions / removals / despawns.
If you do stuff like this, it could explain the jitter.

[echo lion]

Re: history tracking, is this taking into account that updates are unreliable? So you still want history despite the unreliability?

[spring raptor]

Yep, just to receive all availablle bits of information :)

[spring raptor]

@dire aurora I feel like the following should return true?

let confirmed = Confirmed::new(RepliconTick(1));
assert_eq!(confirmed.get(RepliconTick(1)), true);

Maybe change < to <= in get?

[dire aurora]

Which <?

If new does ::default() then .enable(tick) it should return true on get I think 🤔

Or alternatively it could instnatiate it with the tick and 1 for the mask

Tho I guess technically the mask can be all values before the last tick if that's what you mean 🤔

[spring raptor]

Ah, yes, that's what I would expect :)

[dire aurora]

Whats the logic on the sorting there actually? 🤔

[spring raptor]

You mean crates order? We probably should sort them alphabetically...

When we will have more, maybe we will introduce categories 🤔

[spring raptor]

Actually, the logic is easier your way. Will keep it as is.

[dire aurora]

Yea categories might be a good idea, mixing helpers for registring replication, rollback crates, interpolation crates, visibility crates, etc etc would be a bit of a mess I'd imagine 🤔

[spring raptor]

We probably even can do it now... We will have 3 rollback/interpolation crates (bevy_replicon_snap, bevy_timewarp and yours), 1 visibility (bevy_replicon_attributes) and 2 utils (bevy_replicon_repair and bevy_bundlication).

[dire aurora]

Yea, it's more optimized because it allows reusing more work. It's basically just manually doing what Query would

You can fetch the change ticks which bevy also uses internally. In the case of replicon it's just not compared to the last system run but to the last ack (since changes get sent every update until they have been acked, otherwise the packet could get lost or be missing for far too long on the client)

[spring raptor]

is_added used to detect if Replicate was added. For changes we use is_changed.

Yep, it's because change detection is not a part of archetypes :(

[dire aurora]

Event are pretty much how you'd send some things manually yea ... There's an issue about the system-per-event thing tho: https://github.com/projectharmonia/bevy_replicon/issues/245

[spring raptor]

Yes, I will improve it.
We can even do it in a single system instead of using oneshots. Like we do for components.

It was like this in Bevy itself until recent PR: https://github.com/bevyengine/bevy/pull/12936

Will be available in the next release

You quite unlikely will have a hundred of events. Unless you developing a AAA game 😅

I suspect that you misunderstand me. We currently have a separate system for each event. Yes, it's not optimial.
And I saying that we can have a single system for all events. It will be very fast. I just didn't have time to do it.

[dire aurora]

Events can definitely go fairly quickly, but 100 events still doesn't add up to a large amount of overhead, shouldn't be an issue why developing at least 🤔

Iirc it's about a microsecond for a system that does essentially nothing (like systems that handle nothing or just 1 packet)

[spring raptor]

Yep, that's what I trying to say :) Don't worry, use events, I will fix it in the next release. Not in the upcoming one, but in the next after it, this one is pretty packed already.

[dire aurora]

Which is why bevy's internal events badly needed this optimization. 100 replicon events is pretty hard to reach, but 100 bevy events is basically an empty app 😅

[dire aurora]

I'm already encountering bugs while writing tests for the easy history (predicted history), I guess the authoritative history is probably worse and only works somewhat okay by pure coincidence 😅

[spring raptor]

How are you using it right now? I assume that you accumulate events and insert/update this as a component?

[dire aurora]

pub(super) fn update_confirmed(
    mut q: Query<&mut Confirmed>,
    mut events: EventReader<TickConfirmed>,
) {
    for event in events.read() {
        let Ok(mut confirmed) = q.get_mut(event.entity) else {
            continue;
        };

        confirmed.enable(Tick(event.tick.get()));
    }
}

[spring raptor]

Got it!
Do you care about change detection?

[dire aurora]

You mean Changed<Confirmed>? Not really

[spring raptor]

Yes

Thinking if I should avoid triggering it when a tick is too old

I think that Changed<Confirmed> could be convenient, but if we have multiple bits inside, I'm not sure if it's useful since we don't know which one changed.

How you decide to which tick to rollback?

[dire aurora]

Atm it's hardcoded, ideally we'd have a value somewhere that tracks the oldest tick received across the whole app tho 🤔

If we have both the newest and oldest tick received every frame we could probably use that for a bunch of different crates to decide on timing based info ... Buffering would probably want to keep the oldest tick received after the tick it's showing 99-99.9% of the time

[spring raptor]

One more question. What is the difference between confirmed tick and missing one for you?

[dire aurora]

If it's confirmed I know previous data is still valid, if it's not confirmed I can't make that assumption

[spring raptor]

Sure, I understand that, but what is the different in your logic when you can or can't assume that?

[dire aurora]

Mostly just traversing some of the histories backwards (or if the tick that's being resimulated is after history, even looking at the history at all)

I'd imagine it could be used in other ways too, like deciding on if something that was sent for an entity some time ago needs to be undone if it was confirmed or long enough ago (which would also get considered confirmed with the code I sent before iirc)

[spring raptor]

@dire aurora please, try this branch instead: https://github.com/projectharmonia/bevy_replicon/pull/249

It's basically what you suggested. All tests pass, but still in draft because I need to polish some internals.

But I wanted to give you access to it early because of the mentioned mistake in events.

[spring raptor]

This one ^

Since you working on your own tests, it's better for you to rely on the correct replicon part instead of that nasty hack from me 😅

[spring raptor]

@dire aurora Are you interested in an update if it's older then 64 ticks? I.e. can't be represented in Confirmed.

Fixed wording 😅

[dire aurora]

Not really, even in the hypothetical worst case scenario rollback would only be 50 ticks. And that's with a 256 tickrate app that supports up to 400ms ping, I don't think anyone should ever do that 😅

[spring raptor]

I just thinking about not triggering change detection for Confirmed if it doesn't change. And started to think what should I do if history is requested and the update is too old.

[dire aurora]

I guess we could generally just ignore anything that's significantly old 🤔

[spring raptor]

I previosly ignored everything older then the current tick, but since we have optional history, I will ignore everything older then 64 ticks.

[glacial ridge]

has anyone try to use bevy_replicon with godot as a client?

[spring raptor]

I don't think that it's possible. You need something more low-level, like https://github.com/lucaspoffo/renet

[dire aurora]

It would be possible if it was the godot + bevy_ecs approach ofc

[dire aurora]

Still working on those test ... I always struggle to focus on writing tests ... I'm not sure how you even managed to get over 90% coverage on bevy_replicon 😂

My game project is sitting at 6.44%, and a decent portion of that would be easy to test 🙃

[spring raptor]

It's not easy, take your time :)

[spring raptor]

It's totally fine! My game have even less. I found it's not productive when you developing a game at early stages. You recycle a lot of code.

[dire aurora]

That's true if it wasn't for the multiple crates in my game project doing essential stuff ... The input queue is separated out but has barely any tests, it's also currently broken because of some edgecases that happen during startup and then the broken state propagates for a second before it finally works again

[spring raptor]

In my game some essential stuff even not separated 😅

[dire aurora]

Oh yea I have that too, my time syncrhonizatin lives in 5 different files, and the shared part only registers the events cause that needs to be shared between apps 😂

[spring raptor]

Totally normal game development process!

Replicon was part of my game and also barely had any tests. And it was kinda shitty 😅 I wrote tests later and iteratively improved it over time. 24 releases and still have plenty of things to do.

[dire aurora]

I finally found a major bug ... Writing old authoritative values to history doesn't even work correctly 😅

[dire aurora]

Got all my tests passing, and did a quick test (still on the old hacky events thing tho) and I have 0 jitter except when the server actually disagrees with the client (atm only when I activate an ability because the input queue is broken, or when the player dies, cause the client doesn't add that status)

Which means that: 1. There is no jitter; 2. It does still listen to the server

I have not achieved this combination before when any amount of ping latency was present 😂

The non-predicted entities are still very jittery tho, I should really fix the input buffer and pass them trough to clients so enemies can be predicted 😅

[dire aurora]

Switched to the confirmed branch ... Can we get a pub function that calls set with maybe a check to stop people from hitting the debug_assert and a Default impl (or maybe a fn none() -> Self function) that gives a value where no tick is enabled? I need them for my unit tests

[glacial ridge]

does bevy_replicon support replicate to a group of clients instead all of them?

ah I see the bevy_replicon_attributes crate

[glacial ridge]

it seem not meet my requirement 😭

the ClientVisibility is not aware by component right? If I set it false that mean all components will not be replicate right? Is there anyway I can change visibility of specific component?

[spring raptor]

🤔 I can export it... But I thought that you using your own storage and just update it from replicon. If yes, maybe export mask? Or would you prefer to use Confirmed directly for tests?

[spring raptor]

We provide a barebone, but very fast interface to toggle visibility per-entity. This way it's zero-const for games that don't need visibility.
You can use bevy_replicon_attributes for a more convenient interface that just an abstraction on top. Or create your own abstraction.

Per-component is planned, but not available right now :(

[dire aurora]

Your confirmed is nearly identical with my confirmed, would be weird to copy it over 😂

[dire aurora]

No per component visibility yet, the plan is probably going to be having layers for visibility, similar to how collision layers work in physics engines ... Would that work for your usecase?

[spring raptor]

Got it!
But are you sure that you need a constructor? It will be inserted automatically after replication.

[dire aurora]

I need to construct the type to test functions that rely on it individually ... The code that actually uses Confirmed just takes a tick and &Confirmed, so no entities or even world necessary to test them

And I think there's also a test that spawns an entity with it, and ignores the rest of the things that would normally be there cause they don't influence the test (Replicated, map entiries, Predicted, etc)

[spring raptor]

Got it, will make it public. I think it will be convenient for my integration tests too...
One last question about it. You said that debug asserts will cause you inconvenience? I put it mostly because I planned to use them internally and already checked their condition before calling. Also resizing backwards won't work I think...

[dire aurora]

Rather than the debug asserts causing problems, it's the fact that set assumes you wont pass in old values, which you probably can't assume externally. But checking it every time and branching on it would make replicon's internal use different, which is why I suggested another method that just returns if it's old

[spring raptor]

Ah, makes sense

[dire aurora]

I doubt my tests would fail on it, but I can imagine others writing tests running in to that

[spring raptor]

I agree

[spring raptor]

Ah, I know why I don't check it inside set. It's because I trying to avoid triggering change detection.

Or are you suggesting to export another methods that just sets RepliconTick or resizes automatically?

Hard to guess how it will be using externally without the code :)

[dire aurora]

Yea you could make a separate version of set that just checks if it's too old before doing anything ... Like a checked_set or something 🤔

For tests it shouldn't really matter if people always trigger change detection I'd imagine

It doesn't need to actually resize backwards tho, setting a tick in the past to true is basically a no-op if ticks before the mask always return true

If you want to set the ticks back you can just make a new one

[spring raptor]

Just to clarify, are you suggesting to create a function similar to what you had in your version, where you pass actual tick and it resizes if it's bigger or sets a value in history if it's lower?

[dire aurora]

Yea similar to the enable I had. It just sets the bit to true, or does nothing if it would already be true to being out of history

Ah right looking at the code set and resize_to are already separate unlike enable which grouped all operations in one function with a few branches

Slightly more than enable even, since enable would panic if the value was too old 🙃

[spring raptor]

But you didn't panic here

Not sure if I get you... Are you suggesting to export resize_to and an alternative to set that doesn't panic if the value is too old?

Maybe just export set as is? Since you pass an integer instead of tick to set. I would imagine that it would be more convenient for users. Users can easily check if it's bigger then 64.

Want to confirm a tick that 4 tick old since the current? Just pass 4.

Pushed a commit, what do you think about this API?

[dire aurora]

1 << ago will overflow if ago is >= bits

I think it would look something like this?

    pub fn enable(&mut self, tick: Tick) {
        if tick > self.last_tick {
            self.resize_to(tick);
        }
        let ago = self.last_tick - tick;
        if ago < u64::BITS {
            self.set(ago);
        }
    }

enable wouldn't be a great name tho since it would be a less efficient method that only really makes sense for tests that don't care how it functions internally

[dire aurora]

Working with relative ticks makes the tests pretty messy, introducing a pretty significant chance the test passes while doing something entirely wrong

[spring raptor]

Ah, right!

[spring raptor]

How about to call it confirm?

[dire aurora]

Ah yea that makes a lot of sense actually 😅

[spring raptor]

Done! I also renamed get into test. It's more expected name when you work with bits.

Or maybe it will be better as contains?..

[dire aurora]

contains makes sense yea 🤔

I wonder if we should also add something to check for a range of ticks, I do that a lot in my code, but not in the most optimal way 🤔

Also just realized I forgot an edgecase because it failed when I switched to ::new(RepliconTick::default()) instead of my old ::default() ... Old values just get considered valid for all eternity 😅

[spring raptor]

Ah, yeah, I needed to handle overflows

It will consider a tick from u32::MAX as older then 0

[spring raptor]

Feel free to suggest the code.

[spring raptor]

I.e. the logic is a little bit different then the original.

[dire aurora]

My brain is too fried right now to fully comprehend bit operations, but basically my current approach is (previous_tick..last_value).any(|t| confirmed.get(t)) (where previous_tick is the value I'm loading, and last_value is the tick for the next value in history) ... I think it should be possible to replace that entirely with a bitmask operation rather than this loop of calling get/contains ... The current check is (self.mask >> ago & 1) == 1 ... I think if we use the correct ago for the range, use a mask matching the number of bits in the range rather than 1, and change the check to != 0 it would check a range ... And ofc the early return would need to be different too

And I guess the function signature would be something like fn contains_any(&self, start_tick, end_tick) -> bool? 🤔

[spring raptor]

Okay, I will provide it!
Will be away from the computer now, but I will back soon.

[spring raptor]

@dire aurora done!

[spring raptor]

Sorry, made accidental mistake, pull the latest commit, please.

[dire aurora]

I somehow didn't hit that one cause I only use contains_any currently 😂

Everything seems to work (and all my tests pass, including the ones I added earlier), but when I change areas I now get this panic:

thread 'main' panicked at /home/nisevoid/gamedev/bevy_replicon/src/client.rs:333:14:
all init entities should have been spawned with confirmed ticks

[spring raptor]

Interesting, looks like you received an update for an entity that doesn't have Confirmed.
I wondering how it's possible... When I spawn an entity from a mapping or init, I automatically insert Confirmed.

[dire aurora]

When I switch instances I do despawn everything that wasn't there before I connected, could that be related?

[spring raptor]

On client or on server?

[dire aurora]

Client, as far as the server is concerned it just disconnects then on another process it connects

[spring raptor]

I think I know what is going on...
I forgot to clean ServerEntityMap after despawns. Never noticed it before because I didn't do reconnect. I do have tests for reconnect, but I didn't test replication logic after it 😅

Wait, no, I clean it up...

Could you check if disconnect is triggered?

[dire aurora]

I do see it pass trough Disconnected, Connecting, then Connected

2024-05-07T23:37:52.838792Z TRACE bevy_replicon::client: applying update message for RepliconTick(513)
2024-05-07T23:37:52.855678Z DEBUG bevy_replicon::client::replicon_client: changing `RepliconClient` status to `Disconnected`
2024-05-07T23:37:52.892960Z  INFO client::connect: Trying to connect to [::1]:56042 from Ok([::]:43187)
2024-05-07T23:37:52.894859Z DEBUG bevy_replicon::client::replicon_client: changing `RepliconClient` status to `Connecting`
2024-05-07T23:37:52.894887Z DEBUG bevy_replicon::client::replicon_client: changing `RepliconClient` status to `Connected { client_id: None }`
2024-05-07T23:37:52.895268Z TRACE bevy_replicon::network_event::client_event: sending event `net_sync::GetTime`
2024-05-07T23:37:52.938560Z TRACE bevy_replicon::client: applying init message for RepliconTick(5)

It crashed immediately after that last line

[spring raptor]

Strange. You mentioned that you keep some entities, right?

[spring raptor]

Going to sleep now, will be able to help you debug tomorrow.

[dire aurora]

I keep entities from before connecting but that's basically all non-gameplay stuff, some UI entities for example

[spring raptor]

Shouldn't cause any issues... I will try writing a test where I spawn an entity, replicate, disconnect, connect and replicate again, maybe I will be able to reproduce.

[spring raptor]

@dire aurora can't reproduce on my side. Could you take a look at this test, maybe you spot what is different in your usage?

#[test]
fn change_connection() {
    let mut server_app1 = App::new();
    let mut server_app2 = App::new();
    let mut client_app = App::new();
    for app in [&mut server_app1, &mut server_app2, &mut client_app] {
        app.add_plugins((
            MinimalPlugins,
            RepliconPlugins.set(ServerPlugin {
                tick_policy: TickPolicy::EveryFrame,
                ..Default::default()
            }),
        ))
        .replicate::<DummyComponent>();
    }

    server_app1.connect_client(&mut client_app);

    // Spawn user non-replicated entity.
    client_app.world.spawn_empty();

    // Spawn replicated entity.
    server_app1.world.spawn((Replicated, DummyComponent));

    server_app1.update();
    server_app1.exchange_with_client(&mut client_app);
    client_app.update();

    let entity = client_app
        .world
        .query_filtered::<Entity, (With<Replicated>, With<DummyComponent>)>()
        .single(&client_app.world);

    server_app1.disconnect_client(&mut client_app);

    // Despawn the replicated entity.
    client_app.world.entity_mut(entity).despawn();

    server_app2.connect_client(&mut client_app);

    // Spawn a new replicated entity on other server.
    server_app2.world.spawn((Replicated, DummyComponent));

    server_app2.update();
    server_app2.exchange_with_client(&mut client_app);
    client_app.update();
    server_app2.exchange_with_client(&mut client_app);

    client_app
        .world
        .query_filtered::<Entity, (With<Replicated>, With<DummyComponent>)>()
        .single(&client_app.world);
}

What you see should never happen, this is why I used expect :) But apparently there is a bug somewhere.

[grave yarrow]

Hmm, @dire aurora I don't see why fixedupdate would cause noticeable jitters when dealing with prediction

I had some issues a while back with my old client prediction crate of slight jitters

idk I always just thought it was related to some physics bugginess on my part

[spring raptor]

Not sure if you saw, but you may be interested in reading this thread: #networking message

[grave yarrow]

I wish discord had a bookmarking feature or somethin

but ya thank you @spring raptor!

interpolation should only really matter for a couple components I think too then

mainly stuff related to positions

[grave yarrow]

I'm guessing the main way to implement rollback with a custom write fn is to make a custom Command?

since I wouldn't want the deserialization to actually write to an entity, I'd want it to write to a snapshot of components that the server is updating

[spring raptor]

I would suggest to write components to an entity via something like Snapshots<T>.

[grave yarrow]

Ya, I moreso mean the construction of that Snapshot<T> means I need to interrupt the normal replication to an entity and update the snapshot instead.

[spring raptor]

That looks like this:

struct Snapshots<C: Component>(Vec<(RepliconTick, C)>)

[spring raptor]

Yep, you most likely want to start applying the received data after the replication.

[grave yarrow]

I'm assuming I should do all this via the WriteCtx::commands though right?

I don't see much of another way to access Snapshot<T> aside from making a command to do so

This is probably a redundant question, I'm just trying to make sure I'm not overlooking something

[spring raptor]

You have access to EntityMut in a write function too.

[grave yarrow]

ohhhh I see what you mean

Snapshots<T> as a component not a resource

[spring raptor]

Yeah, it's much more convenient this way.

[spring raptor]

It's okay, feel free to ask anything.

Improved the docs to make it more friendly for beginners in networking:
https://github.com/projectharmonia/bevy_replicon/pull/250

[grave yarrow]

the RemoveFn part is a bit more awkward to deal with here

[spring raptor]

Ah, you probably need access to an entity?

[grave yarrow]

it'd make it a bit simpler, but it still is fine the way it is, I just have to do it this way:

pub fn rollback_remove_fn(ctx: &DeleteCtx, mut entity_commands: EntityCommands) {
    entity_commands.add(|entity: Entity, world: &mut World| {
        if let Some(mut snapshot) = world.entity_mut(entity).get_mut::<Snapshot<C>>() {
            snapshot.remove(&ctx.message_tick);
        }
    });
}

actually wait this won't work hmm

since I'd need a generic here...

[spring raptor]

I think that you provide it when you register your funciton

[grave yarrow]

yeah, was just thinking that

[spring raptor]

I.e. just put generic to the function itself and instantiate it

About entity access, I will provide it. I fetch the entity for replicon's internal logic anyway, so I will just pass it to the RemoveFn.

[grave yarrow]

this works:

pub fn rollback_remove_fn<C: Send + Sync + 'static>(ctx: &DeleteCtx, mut entity_commands: EntityCommands) {
    let tick = ctx.message_tick;
    entity_commands.add(move |entity: Entity, world: &mut World| {
        if let Some(mut snapshot) = world.entity_mut(entity).get_mut::<Snapshot<C>>() {
            snapshot.remove(&tick);
        }
    });
}

[spring raptor]

You can replace Send + Sync + 'static with Component

[grave yarrow]

ah true

[spring raptor]

I wish I could add the type to the function signature. I.e. make it generic (RemoveFn<C>), like I did with writing. It would be much more ergonomic. But there is nothing I can attach the type to :(

[grave yarrow]

pub type RemoveFn<C> = fn(&DeleteCtx, EntityCommands, compulsive: PhantomData<C>);

hehe

[dire aurora]

Yea it's not really prediction that's the source of jitter in the first place. If you have good rollback prediction causes no jitter. But FixedUpdate still causes jitter by itself, and if you do have interpolation but have some mismatch with your FixedUpdate you also get jitter. I think that issue mostly covered this latter case

[dire aurora]

Hmmmm that's strange ... Maybe it's not just an entity directly getting spawned by the server? There's also things that get spawned from mapping and maybe the predicted entities mapping also works differently 🤔

[spring raptor]

Yeah, I thought about it, but looks a bit awkward in for my taste :)

[spring raptor]

I insert it for predicted spawns too...
https://github.com/projectharmonia/bevy_replicon/blob/1614042a193f810b974f6ddc951a70f72b4bd90e/src/client.rs?plain=1#L293
But I will try in the test just in case.

Nope, no panic...

[spring raptor]

Maybe you connecting again in the same tick?

[dire aurora]

Disconnected, Connecting and Conncted are all spaced apart exactly one frame

[spring raptor]

Do you insert mappings into ServerEntityMap?

[grave yarrow]

I'm a bit conflicted on whether to use FixedMain directly or not

internally I feel like it'd be better to have a separate GameMain schedule

that just runs at the same cadence but also rolls back + replays

[dire aurora]

If you're doing rollback make a schedule that contains the entire simulation, then you can just execute that in FixedMain, and also in the schedule for rollback

[grave yarrow]

yeah, just trying to weigh the options, since doing it that way does add some burden to people using it

then again, search and replacing your code for FixedUpdate -> GameUpdate isn't too big of a deal

[dire aurora]

I just made my rollback system configurable

app.add_plugins(RollbackPlugin {
    store_schedule: SimSchedule::Simulation.intern(),
    rollback_at: PostTransition.intern(),
    rollback_schedule: SimSchedule::Resimulation.intern(),
})

I like how every schedule I used there is custom

[grave yarrow]

typing out app.add_systems(SimSchedule::Simulation, ...) probably gets old though lol

[dire aurora]

It would but luckily I don't have to do that

Cause every plugin is configurable

Since the server needs to run them in Update, while the client runs them in SimSchedule::Simulation 😂

I should probably just shorten those names at some point tho

Really would be nice if I could just turn this rollback into a crate already without people needing to use a custom bevy branch 🙃

[dire aurora]

Tho honestly even if it requires the custom branch it might be worth making a crate for it already ... Getting rollback right is an absolute nightmare, especially if more things need to be rolled back than just 1 entity

Lightyear doesn't do prediction that way I think, and there were some subtle issues with bevy_timewarp ... Making something that just does everything the right way from the start, even if that requires custom bevy branches seems like the most viable solution if we ever want to get this problem solved 😂

Still need to add component disabling to bevy to make the solution fully correct tho

[grave yarrow]

Why do you need a bevy branch for it?

[dire aurora]

Disabling entities, and later components

[grave yarrow]

are you disabling them for performance or a more functional reason?

I'm workin on just a very naive rollback replay thing rn without much optimization

[dire aurora]

Both ... Disabling them is the faster route, but you also can't spawn fully new entities. So the workaround would involve clearing the entire entity then filling it if re-enabled, but systems could still detect that in a wrong way and decide to remove the entity because it's invalid

For the most part things like rollback and networking are easier with ECS, but entity ids do present some annoying issues

[grave yarrow]

Can't spawn fully new entities?

[dire aurora]

If you spawn new entities when re-enabling them, every reference to that entity will be broken

It would be annoying enough to fix that for just the present, but you would need to fix it for every value in the predicted and authoritative histories 🙃

[grave yarrow]

is this a situation of spawning/despawninf?

I don't understand the re-enable part of this

[dire aurora]

When something got "despawned" a frame ago, and you roll back to before it got despawned, it needs to exist again

If you actually despawned it your simulation will be wrong

[grave yarrow]

tbh I'd probably just solve that by saying you don't ever truly despawn an entity, you mark it for despawning and that happens after the rtt / 2 buffer

[dire aurora]

That's where disabling entities comes in

You can't just leave them around, then they would also break your simulation

[grave yarrow]

I suppose ya

bevys despawning in general is a bit of a nightmare tbh

is disabling entities coming in the next bevy version?

or just unknown

[dire aurora]

I made a PR for it but it got pushback for essentially no reason, and the alternative approach that unblocks the usecase has been sitting on X-Controversial getting no real attention 🙃

[grave yarrow]

What are the prs for those?

[dire aurora]

https://github.com/bevyengine/bevy/pull/12928 and https://github.com/bevyengine/bevy/pull/13120

[grave yarrow]

Mobile is a pain, but I could search em up i suppose

Ty

[dire aurora]

We'd also need this for component disabling, which requires this PR that has been stuck in a review abyss for some reason too ... https://github.com/bevyengine/bevy/pull/11475

But yea ideally the two PRs that unblock things get merged and we can at least get something working, even if very hacky in 0.14 ... Then during the cycle for 0.15 we could make a workgroup for disabling. Many networking usecases would probably benefit from having disabling in general

[dire aurora]

Also not really saying there are no issues with the Disabled approach, I think we'd really just need to experiment with it before making another PR 😂

[spring raptor]

You can get attention by mentioning it again in #ecs-dev. Another idea would be showing something cool in #showcase and saying that it depends on you branch :)

[dire aurora]

I'll probably try to mention joy when they are around, since pinging the SME role seems pretty ineffective in general

[spring raptor]

Another option would be publishing the crate and mention that it requires a branch beacause of the PR.

[spring raptor]

I personally just jump in with something like "Can I have more opinions on #?" 😅

[dire aurora]

#ecs-dev message

Like that one? 🤔

[spring raptor]

Damn 🤣

Well, another message won't hurt... Sometimes it works from a second attempt :)

[spring raptor]

Do you insert entities via ServerEntityTicks?

[grave yarrow]

@dire aurora the latter one at least seems to be slowly coming together more

I'll look at giving it a review soon

[dire aurora]

I never mutably access it, idk if there's some other way things get added to it tho 🤔

[grave yarrow]

The DefaultQueryOptions seems pretty clean

[dire aurora]

Yea I just took the non-controversial parts out of Disabled to unblock the usecase

Then alice put the controversial label on it 🙃

[grave yarrow]

tbf thats put on quite a lot of things that are actually beneficial

Id say my FixedUpdate gizmos stuff was more genuinely controversial

I don't think this one is really that controversial tbh

[dire aurora]

I think it would've been labeled contentious instead of controversial if I opened it after the label change 😂

[glacial ridge]

Hi, I'm trying to create a game like Among Us, each game will be run on 1 thread. The renet server will run on main thread and pass data to games via channels. The problem when I trying to use bevy_replicon is the thread dont have access to the renet server. Is there anyway I can listen to the ReplicateEvent for example, which will contain all data that I need to send back to the renet server on main thread and then to the client?

[dire aurora]

Does each thread run its own app?

[glacial ridge]

yeah each thread run 1 bevy app

[dire aurora]

I guess you could implement your own version of the logic from bevy_replicon_renet 🤔

[glacial ridge]

Ah I see, all the data is in RepliconServer, thank you.

[spring raptor]

@dire aurora I removed the panic, it should print the error with entity ID instead. Could you take a look at this entity? Maybe it will give me a clue about what is going on.

[dire aurora]

Getting at least 50 of them, tho I can't tell you much just based on the IDs ... Also getting another panick from a later line now:

thread 'main' panicked at /home/nisevoid/gamedev/bevy_replicon/src/client.rs:441:14:
all entities from update should have confirmed ticks

[spring raptor]

Hm... Looks like mappings are all wrong...

Do you have an inspector?

Maybe by looking at the ID you will be able to detect what kind of entities are referenced instead of the replicated?

[dire aurora]

I don't, tho we could probably just log the components in the warning (similar to what commands.entity(e).log_components() does)

[spring raptor]

Done, could you try again?

[dire aurora]

They're all the same:

2024-05-08T20:17:53.673109Z ERROR bevy_replicon::client: entity without Confirmed: 80v3
2024-05-08T20:17:53.673131Z  INFO bevy_ecs::system::commands: Entity 80v3: ["bevy_replicon::core::Replicated"]

[spring raptor]

This is weird... So we have multiple entities with Replicated, but without Confirmed...

[dire aurora]

Yep

[spring raptor]

Ah, I found the problem!

It's the mapper, I forgot to insert Confirmed in the mapper

Thank you a lot!

[dire aurora]

My game is very good at finding bugs in all libraries I use 😂

[spring raptor]

@dire aurora will be a little bit weird for ComponentMapper and ComponentWorldMapper, you will need to pass a tick to them...
Maybe it's better to remove them? I feel like that use case is quite rare, users can create their own versions with just a few lines of code.

[dire aurora]

Ideally we would be able to just have unconfirmed Confirmed

Because things spawned by mapping are never confirmed

[spring raptor]

I can turn it into an option, but it's not needed for replicon, I force-override with everything I receive from init messages.

[dire aurora]

A mask of 0 should also work for unconfirmed right? 🤔

[spring raptor]

Probably a good idea, I will rework the logic then. And provide a constructor for unconfirmed.

[spring raptor]

Makes the API too tricky to use. Like last_tick() will return default value.

Probably better to go with Option instead...

[dire aurora]

last_tick returning 0 doesn't sound like too big of an issue tbh 🤔

[spring raptor]

Comparsion wraps. 0 could be bigger then u32::MAX / 2 + 1.

[dire aurora]

Oh ... Why does it need to wrap? Has anyone ever had their tick hit u32::MAX?

[spring raptor]

Unlikely... 🤔
But maybe it worth to switch to u16 to save bandwith.

[dire aurora]

Ah yea it would make more sense with a u16

[spring raptor]

Will do it in a separate PR.

Another option for Confirmed would be to back to HashMap. But It's slower to access. Maybe I better go with Option.

[dire aurora]

Yea Option seems reasonable

The component is also easier to use than accessing a map

[spring raptor]

Another idea: rework the logic to insert the component on actual confirmation.

Having to call unwrap each time will be ugly.

[dire aurora]

That would also work ... It's a bit slower but it would remove this edgecase

Not that spawning entities is fast in the first place 😅

[spring raptor]

Not sure if it's even slower... I access this component, so instead of expect I can just else into a command...

[dire aurora]

The command would still be slower because of the archetype move

But only when the entity just got spawned and doesn't have it yet

[spring raptor]

I assume that we will have command batching in the future :)
But yeah!

[dire aurora]

Yea hopefully this will be a non-issue in the future

Probably better to go with the safer option here, panicing on a missing component seems a bit silly

[spring raptor]

Agree

[dire aurora]

People could also remove it for whatever reason after all

[spring raptor]

@dire aurora could you try it again?

[dire aurora]

That u16 change would actually probably be decently complex, since systems would expect to receive valid message ticks ... Replicon could probably just sync them to be in the same u16 cluster when connecting tho ... Tho maybe ideally we'd have a more advanced system that turns the received tick into some useful simulation time (the tickrate might not match the replication rate after all) ... Would kind of mean we need a universal simulation time tho

[dire aurora]

It works now 👍

[spring raptor]

But why so? It should wrap correctly.

[spring raptor]

Great, marked as ready for review for koe.

[dire aurora]

It would wrap correctly but if the game receives u16s it can't be used to save bandwidth by giving values relative to that tick anymore

[spring raptor]

Ah, I see.

[dire aurora]

There's also a bunch of edge cases to handle like contains_any where the first tick is higher than the right tick because the right one wrapped

[spring raptor]

Wrong link :) For some reason my plugin links to koe's fork

https://github.com/projectharmonia/bevy_replicon/blob/ae6ad8178200ec032fabb94a4e14bd31301da8b1/src/server/replicon_tick.rs?plain=1#L44-L51

[dire aurora]

Oh so that's what that weird comparison was for, I was wondering why that was in there 😂

[spring raptor]

So yeah, you can run server for more then 2 years and it will work 😅

[dire aurora]

Overall it probably wouldn't be too bad to have replicon be a bit more aware of simulation ticks tho, it would just be annoying currently because the bevy ecosystem has very few features to actually support things like networked simulations well 😅

[dire aurora]

Except my own tick type would still overflow and break things 🙃

[spring raptor]

What do you mean?

[spring raptor]

I think you can still use your trick, u16 is quite big...

[dire aurora]

If replicon sends a u16, knows how its ticks maps to the simulation clock, it can just get the correct simulation tick that u16 would map to

[spring raptor]

Under simulation clock you mean Tick?

From Bevy

[dire aurora]

Not quite, bevy's Tick is just a meaninless value that increments at a really high rate ... In bundlication I have another Tick value (just to make things more confusing!) and it's supposed to be incremented every time the simulation runs, we'd basically need somehting like that but in some universal place ... Like Time<Simulation> or something

It could simply be a u64 counter and a delta

Would be much nicer if replicon could then pass in the value that message would've been on that simulation clock, rather than a tick value that might not even match up because you might tick 60 times per second but only replicate 20 times per second

[spring raptor]

Aren't you increment RepliconTick each simulation?
It's basically what it is.

[dire aurora]

I do, but not every user would necessarily do that

[spring raptor]

We provide some quick configuration, but for rollback you definitely should use TickPolicy::Manual.

[dire aurora]

But if we get a u16 value I already can't map the RepliconTick to my own Tick, since I wouldn't have a clue which simulation tick it actually maps to

[spring raptor]

But why do you need your own tick? 🤔

[dire aurora]

Same thing applies for anyone that only increments tick every 2 or 3 simulation ticks

[dire aurora]

Because RepliconTick can't turn back and isn't made to do tons of gameplay math

In my game almost everything runs on ticks, since durations are pretty useless when you only increment by a fixed delta

[spring raptor]

Maybe I can provide some necessary abstractions? Or you think that it will be abusing the type?

I mean the necessary logic for RepliconTick

It's better to have a convenient ecosystem

[dire aurora]

This is kind of why bevy needs to get a simulation tick, it's pretty silly that any third-party crate that does not specifically use replicon or lightyear or something like it has no access to this sort of timekeeping

[spring raptor]

Yeah...

[dire aurora]

But because FixedUpdate still has issues the only people really using FixedUpdate are people with networked or physics based games

The rest just uses variable deltas and then using durations for things makes sense

[spring raptor]

I agree. But what would be the best short-term course? I think we need to make writing logic for replicon convenient.

This is why I suggesting to add some methods that are necessary for rollback plugins.

Like with Confirmed.

Not directly related to rollback, but useful for it.

Also having the ability to decrement would simplify my integration tests...

[dire aurora]

Yea it's a pretty weird thing to do tho ... I did allow it in bundlication but it would be pretty bad if the server actually ever rewinded it 😅

[spring raptor]

Hm... True.

[dire aurora]

I think in the short term we can just get optimizations like u16 ticks by just tracking what the "real tick" would've been as a larger type that doesn't wrap

Tho it's hard to call anything a "real tick" given there's no actual source of truth to be found outside user code 😅

[spring raptor]

But why? Maybe just add wrapping support to your tick? It's simple. You can even copy the logic.

[dire aurora]

You can't wrap the tick if you use it for timekeeping

Like if I make a buff that lasts 5 hours, I just set it to the tick that happens in 5 hours

So the current tick + 180000

[spring raptor]

But why you need such a big buffer?..

Do you care about such an old data?

[dire aurora]

In some cases where things look at the past I generally ignore old data, like the last time the player touched the group clamps at 255 ticks

[spring raptor]

One thing to consider - user still can insert RepliconTick::default()...

[dire aurora]

But the value is still relative to the current tick, which is an counter that started when the server started

[dire aurora]

That's true, but that feels a whole lot more wrong than using a provided decrement_by method or using a DerefMut to do -= 1

[spring raptor]

Yes, I understand it. But if we just switch to u16, there won't be any difference, unless you want to track something older then 65535 ticks.

[spring raptor]

The last message was about this ^

[dire aurora]

The problem comes in when you do things like this:

fn read_new(mut r: impl std::io::Read, ctx: &mut DeserializeCtx) -> BincodeResult<Self> {
    let mut b = [0u8; 1];
    r.read_exact(&mut b)?;
    Ok(Self(Tick(ctx.message_tick.get().saturating_sub(b[0] as u32))))
}

This type contains Tick, which is just my game's time, so if the RepliconTick wraps this would always produce incorrect time values

Ideally we would have a universal SimulationTick in bevy, and both my type and ctx.message_tick would use that as values, but then internally replicon can network things however it wants because it's not that hard to figure out in which block of multiple minutes the time currently falls

[spring raptor]

But why use saturating_sub here? Why not just wrapping_sub?

[dire aurora]

That wouldn't help, since b[0] is just a u8 value relative to the Tick

[spring raptor]

And switch to using u16 for your tick

[dire aurora]

This is that part that wouldn't work. Now any thing in my game's logic can only be 18 minutes long

[spring raptor]

That's what I asking, do you need anything longer then 65535?

Ah, I think I get it. You mean that you use everything in ticks?

Like you can have a buf that lasts for an hour

[dire aurora]

Yea, everything uses ticks. A skill that lasts 5 seconds uses ticks, a buff that lasts multiple hours uses ticks, the time since you last touched the ground uses ticks

This might look like a real duration, but it's also ticks 😅
cooldown_duration: TickDuration = ms(5000);

[spring raptor]

Ah, I see, cool idea!

[dire aurora]

The main reason is that using timers makes a lot of things non-deterministic

[spring raptor]

Makes total sense

[dire aurora]

As a result the Tick type on bundlication (which was just copied over from my game's logic) ended up quite bloated https://github.com/NiseVoid/bevy_bundlication/blob/main/src/tick.rs 😅

Oh and I didn't even push the ugly impl From<Tick> for RepliconTick yet 😂

I could probably get rid of that if I just use replicon tick directly in rollback tho, which I probably should because my rollback crate doesn't need to depend on bundlication ... It doesn't even replicate things

[spring raptor]

Opened an issue about it: https://github.com/projectharmonia/bevy_replicon/issues/251
I agree with the suggestion. Probably something like 2 u16 would work, one for generation and another one for tick. If the difference is bigger then u16::MAX / 2, then overflow happened and we can just increment the generation.

[spring raptor]

Why do you need the tick in bundlication now?

[dire aurora]

No real reason, it's just tech debt 😅

[dire aurora]

I would probably need math ops to do this while wrapping correctly ... Maybe we should separate the tick type from the resource that tracks the tick the server is at? 🤔

The resource could just not expose anything besides increment(_by), and no public way to construct it either, while the tick type can provide regular math ops and make sure everything wraps correctly

[spring raptor]

Probably not a bad idea... One issue is my integration tests, I sometimes copy the original tick and restore it back.

[dire aurora]

Copying a value would still be possible, but I don't think many people would do that to mess with the tick

[spring raptor]

Will think about the API tomorrow, it's late night for me.

Glad that we figured everything else. After the tick change I probably could draft a new release.

[dire aurora]

The u16 optimization or the split between wrapping type and resource? The optimization we could probably push back further ... But not being able to do math on the tick type would probably be something people will hit since replicon exposes it to the user in more places now 🤔

[spring raptor]

I thought both 😅
I was thinking about making MessageTick (maybe not the best name) u16 that wraps and have all operations and RepliconTick that stores generation and current MessageTick

Maybe not the best idea, just thinking out loud. Going to sleep, if you think that it might not work, let me know.

Maybe you are right and it worth just do the split for now...

[grave yarrow]

is the u16 even exposed?

I was also thinking about this last week

I guess yeah it is

[grave yarrow]

I think thisd work well, hide the underlying wrapping behavior and increment on the clients/server as if they were u64s or something

Might even be able to get away with u8 at that point

[dire aurora]

Unlikely, a short lagspike can easily delay things by a few seconds, combine that with a decently to high tickrate and you have no clue what was sent when

[spring raptor]

@dire aurora I'm not sure if it will work for your use case.

Because you will need full u32 tick to be passed into write/remove, right?

[spring raptor]

I will do it a bit differently then. I will split the resource and tick just as you suggested first and then I will add the mentioned optimization by sending only tick.get() % u16::MAX that will be converted into a full tick.

[spring raptor]

Also could you explain what tick do you rollback now? You have your own resource with your own tick that you increment/decrement?

[dire aurora]

Yea I have my own tick (technically the one from bundlication) in rollback, but it probably makes more sense to just have things be configurable ... Then I can finally move the bundlication tick back to my own code where it originally came from 😅

[spring raptor]

Got it! So it would be convenient for you if I split tick from the resource that you can store it into your own resource?

Just want to make sure if I understand the use case correctly :)

[dire aurora]

Yea it would definitely be good if the type is easy to use so it can be stored in other resources where the user is more free to mess with the value, or convert it to another tick counter type the user has

There would basically be 3 types I guess: The networked version (which might not even need to be a named type), the "total tick" and the server resource that's supposed to only ever increment

And the total tick should be easy to sync if the server just sends it when a client connects I guess 🤔

[spring raptor]

Even 4, there is also last received init tick from server on client :D

[dire aurora]

Ah right

We need as many Tick types as possible until one day the ecs devs realize that Tick is a really bad name for the change detection thing 😅

[dire aurora]

Finally made an issue about the simulation clock thing ... https://github.com/bevyengine/bevy/issues/13306 ... I wonder if anyone doing non-networked stuff will show up to say they also need it 🤔

[spring raptor]

Any suggestions about the naming? Thinking about RepliconTick for the tick and ServerTick for the resource...

[dire aurora]

That seems reasonable

[spring raptor]

@dire aurora could you check https://github.com/projectharmonia/bevy_replicon/pull/254 ?
Suggestions are welcome.

@dire aurora made small mistake, please pull the changes if you started using 😅

[dire aurora]

Haven't started using them yet, I'm currently trying to fix the broken input queue so I can actually make a showcase of how well rollback works with variable ping without all the mispredictions from the server executing everything later

[spring raptor]

As requested, access to EntityMut: https://github.com/projectharmonia/bevy_replicon/pull/255

[spring raptor]

Actually, it's not that easy. If client doesn't receive insertions / removals / despawns for ~8min, it will break events because ServerInitTick on client will be considered lower...

[dire aurora]

Hmmm ... I think you'd probably want to only ever network u16 ticks, and always compare using a full tick (which could even be made u64, since it doesn't get networked anyway)

That way as long as you receive data often enough you should be able to keep the tick correct ... And I think if you haven't received data in ~8 minutes you probably disconnected 😂

[spring raptor]

Server may not have insertions / removals / despawns, but have component changes. In this case ServerInitTick won't be updated.
This is why I saying just sending u16 and compare to a full tick could break it :(