#bevy_replicon

1 messages · Page 6 of 1

timber grove
#

seems like it would still do the trick

#

are they not synced with client / server replicon ticks?

echo lion
#

RepliconTick only increments on the server when replication diffs are to be checked (you can manually increment it if you want lots of control). On the client, it only increases when receiving an Init message, which are only sent on entity spawn/despawn or component insertion/removal.

timber grove
#

ah okay

#

that makes sense, thanks

spring raptor
#

Merged. Thanks for the edits :)
I will open one more and draft a prepare a release.

#

I think it can be used to detect how old the received update is and apply interpolation. Assuming you send updates on fixed timestemp.
Never mind, looks like we don't send it on updates.

#

So yes, for events you need to include the needed information inside the event. But it's not hard, you can create your own generic wrapper with the timestamp and T for your event and register your wrapper instead of just T.

#

@echo lion we probably should add an option to include tick in every update. It could be quite useful for prediction. What do you think?

echo lion
spring raptor
#

Good point.

echo lion
#

In my projects I always export everything, because fussing with using statements is a pain.

spring raptor
#

You mean that you include everythin in prelude?

echo lion
spring raptor
echo lion
spring raptor
#

Just to follow best practices.
From what I read, people usually put only common things in prelude.

#

Just a suggestion, though. I will close if you don't think that it's the right path.

#

Except some doc examples formatting from it should be useful.

echo lion
spring raptor
echo lion
#

I will look at the PR later today.

spring raptor
spring raptor
#

@granite hill
@willow osprey
Hi! Are you still interesting or planning to continue to maintain bevy_replicon_snap and bevy_timewarp?
Asking because we might have a breakage that could affect your crates.
If you are interested in using bevy_replicon for prediction/interpolation, please let me know :)

granite hill
spring raptor
spring raptor
granite hill
spring raptor
spring raptor
# granite hill Yep, I relied on the RepliconTick for determining which inputs need to be replay...

We splitted our replication channel into two (https://github.com/projectharmonia/bevy_replicon/blob/3b06733d58c38531dad230f547445c1cfa2ac822/src/core/replicon_channels.rs#L9).
First one is for reliable data (like component insertion, removal, etc.) and second one for updates (like when a component changes).

When you receive a component, the same deserialization function called for both reliable and unreliable updates:
https://github.com/projectharmonia/bevy_replicon/blob/3b06733d58c38531dad230f547445c1cfa2ac822/src/client.rs#L441
https://github.com/projectharmonia/bevy_replicon/blob/3b06733d58c38531dad230f547445c1cfa2ac822/src/client.rs#L357
Both functions will receive the current replicon tick. So this part should work as before.

But entities will have different ticks, that's what changed.
If you need to access ticks information outside of the deserialization function, use https://github.com/projectharmonia/bevy_replicon/blob/3b06733d58c38531dad230f547445c1cfa2ac822/src/client.rs#L534
Let me know if have any questions :)

Sorry for a lot of breaking changes, but from now on I think the API will be more or less stable.

spring raptor
#

Preparing the release now :)

spring raptor
echo lion
timber grove
#

just wanted to say I appreciate having most things included in prelude. With bevy there are a ton of features that aren't in prelude that I always need, like SystemId, utils::HashMap, Renderlayers, etc. so it always looks dumb at the top of each file I have bevy::prelude::*, and then like one or two other random structs imported

#

I generally think it's pretty hard to predict which features will be commonly used or not

granite hill
spring raptor
timber grove
#

Hey so, a couple questions about the Client Entity Map example here: https://docs.rs/bevy_replicon/latest/bevy_replicon/server/struct.ClientEntityMap.html
I don't quite understand how that works? Like it looks like it will work fine once the event gets received by the server, but what if the event does not get received by the server until after the server has already sent replication data to the client? Won't there still be an extra entity that gets spawned on the client? if so, does that get cleaned up when the event is received by the server?

#

also in that example, would a EventType::Unordered be best, do you think? EventType::Unreliable would probably be bad because it could miss it entirely, right?

echo lion
# timber grove Hey so, a couple questions about the Client Entity Map example here: https://doc...

Like it looks like it will work fine once the event gets received by the server, but what if the event does not get received by the server until after the server has already sent replication data to the client?

The idea is you send the client entity in the same client event that triggers the server to spawn the predicted entity. This way there are no race conditions even if you use unordered client events.

#

See the client event SpawnBullet(Entity) contains the predicted entity.

timber grove
#

Oh I see. So in my case I'm spawning heros, and each client spawns a hero for each player that's connected to the server, upon their connection. Each hero has some components that are replicated, but most components are not replicated. So in my case I'm not sure how to structure it so that would work

echo lion
timber grove
#

but won't that only spawn the components that are replicated?

#

I guess then I could query for Added<Hero> and then add the rest of the bundle?

echo lion
echo lion
timber grove
#

okay that makes sense, thanks

runic nest
#

Cant compine bevy_replicon with bevy 0.13
rustc 1.76.0 (07dca489a 2024-02-04)

error[E0283]: type annotations needed
  --> C:\Users\u108\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_replicon-0.24.0\src\core\replicon_channels.rs:80:41        
   |
80 |         if self.server.len() == u8::MAX.into() {
   |                              --         ^^^^
   |                              |
   = note: multiple `impl`s satisfying `usize: PartialEq<_>` found in the following crates: `core`, `serde_json`:
           - impl PartialEq<serde_json::value::Value> for usize;
           - impl<host> PartialEq for usize
             where the constant `host` has type `bool`;
help: try using a fully qualified path to specify the expected types
   |
80 |         if self.server.len() == <u8 as Into<T>>::into(u8::MAX) {
   |                                 ++++++++++++++++++++++       ~

error[E0283]: type annotations needed
  --> C:\Users\u108\.cargo\registry\src\index.crates.io-6f17d22bba15001f\bevy_replicon-0.24.0\src\core\replicon_channels.rs:94:41        
   |
94 |         if self.client.len() == u8::MAX.into() {
   |                              --         ^^^^
   |                              |
   |                              type must be known at this point
   |
   = note: multiple `impl`s satisfying `usize: PartialEq<_>` found in the following crates: `core`, `serde_json`:
           - impl PartialEq<serde_json::value::Value> for usize;
           - impl<host> PartialEq for usize
             where the constant `host` has type `bool`;
help: try using a fully qualified path to specify the expected types
   |
94 |         if self.client.len() == <u8 as Into<T>>::into(u8::MAX) {
   |                                 ++++++++++++++++++++++       ~

For more information about this error, try `rustc --explain E0283`.
error: could not compile `bevy_replicon` (lib) due to 2 previous errors
#

any ideas?

willow osprey
#

hi ~ just dusting off bevy_timewarp and upgrading from 0.11 to 0.13. hoping to get back at it. just reading up on replicon changes, cool that the transport is factored out now

runic nest
runic nest
#

strange

#

but in my project it doesn't compile

echo lion
runic nest
#

probabably versions issue

#

how can I check for versions overlapping?

echo lion
#

Looks like you are importing multiple imlementations of PartialEq for usize.

#

Ok the fix should be u8::MAX as usize instead of u8::MAX.into(), can you PR it? @runic nest

runic nest
runic nest
echo lion
runic nest
#

it my first PR ever, I trough I did everything the right way

runic nest
spring raptor
spring raptor
spring raptor
runic nest
#

I'm probably dumb, but I cant import it using bevy_replicon = { path = "../bevy_replicon"}

spring raptor
#

Why so?

echo lion
runic nest
#

trying

#

wait please, I'm facing so many issues

spring raptor
#

It's because serde also defines its own impl for u8.

#

We don't use serde json in our examples, this is why it works for us

#

So we should merge the fix and draft a new release.

runic nest
#

like from 0.23 to 0.24

spring raptor
runic nest
#

I think you should merge it yea

#

I still cant fix a bunch of issues with my migration process but I think that the problem fixed

spring raptor
spring raptor
echo lion
spring raptor
#

@runic nest published 0.24.1

runic nest
#

oh thanks!

#

finaly I did it

#

it works

#

hell yeah

#

migration from bevy 0.12 to 0.13 was not that easy

spring raptor
# runic nest migration from bevy 0.12 to 0.13 was not that easy

Ah, if you were also migrating from 0.12 to 0.13, it would be easier for you to use the previous replicon release while doing so (it supports 0.13). I never do a huge changes before bevy releases to avoid users dealing with huge pile of diffs at once :)
And after everything compiles, migrate to the latest replicon.

#

But glad you migrated :)

timber grove
granite hill
spring raptor
spring raptor
granite hill
runic nest
#

I'm probably missing something. But the player on the client moves way much faster than the player on the server? I'm doing it like this:

#[derive(Debug, Default, Deserialize, Event, Serialize)]
pub struct MoveDirection(pub Vec3);

fn input_system(mut move_events: EventWriter<MoveDirection>, input: Res<ButtonInput<KeyCode>>) {
    let mut direction = Vec3::ZERO;
    if input.pressed(KeyCode::KeyD) {
        direction.x += 1.0;
    }
    if input.pressed(KeyCode::KeyA) {
        direction.x -= 1.0;
    }
    if input.pressed(KeyCode::KeyW) {
        direction.z += 1.0;
    }
    if input.pressed(KeyCode::KeyS) {
        direction.z -= 1.0;
    }
    if direction != Vec3::ZERO {
        move_events.send(MoveDirection(direction.normalize_or_zero()));
    }
}
fn movement_system(
    time: Res<Time>,
    mut move_events: EventReader<FromClient<MoveDirection>>,
    mut players: Query<(&Player, &mut Transform)>,
) {
    const MOVE_SPEED: f32 = 3.0;
    for FromClient { client_id, event } in move_events.read() {
        for (player, mut position) in &mut players {
            if *client_id == player.0 {
                position.translation += event.0 * time.delta_seconds() * MOVE_SPEED;
            }
        }
    }
}

// in the App
.add_client_event::<MoveDirection>(ChannelKind::Ordered)
#

I'm in 3D if it matters

#

And I also have a "copy" of the client that goes before the client. Like a ghost or something

#

I thought that the problem in the TickPolicy, but it's not

spring raptor
#

If yes, then you can't do it this way because you will have different frame delta.
Instead you need to send the movement and wait for server to replicate the position back.

runic nest
spring raptor
runic nest
#

what I should send instead of direction? The new position?

spring raptor
#

See the simple_box example in the repo.

#

But! If your game is fast paced, you may not want to wait for server to replicate the position back. Also the movement won't be smooth.
In this case you need bevy_replicon_snap.

runic nest
#

but It's still unclear. As I see in the simple_box apply_movement system looks almost the same as mine

#

and I register it the same way

.add_client_event::<MoveDirection>(ChannelKind::Ordered)
            .add_systems(
                Update,
                (
                    movement_system.run_if(has_authority), // Runs only on the server or a single player.
                    input_system,
                ),
            );
spring raptor
runic nest
#

I'm not sure about the fact that server is behind the client

#

there's a lags on client

runic nest
#

the code

#

I added .replicate::<Transform>() in the main

spring raptor
spring raptor
runic nest
#

looks like it's pretty simple to integrate

runic nest
spring raptor
#

Figured it out :)

#

@runic nest you did it correctly. And it's a client a little behind the server, not vice versa. So it's expected.
You can set replication to every frame and the mentioned lag will gone.
But for your type of game you definitely need something like bevy_replicon_snap. Yes, it's easy to integrate, try it out.

runic nest
#

thank you!

tardy knoll
#

Hi - what would be a good way to identify the current player only in a system? Comparing player's client_id with NetcodeClientTransport::client_id? Thank you

Edit: I can just add a marker when spawning the player

spring raptor
runic nest
spring raptor
runic nest
spring raptor
#

The client usually always slightly behind in movement because of the latency. But the end position should be always in sync.

runic nest
spring raptor
runic nest
spring raptor
#

Yes. It shouldn't run smooth, but the travel distance should be roughly the same.

#

What OS are you running?
Your problem could be caused by the server slowing down in foreground.

#

And time.delta_seconds() will be bigger.

runic nest
spring raptor
runic nest
runic nest
#

sooo

#

what's the problem with my code and bevy_replicon_snap example??

#

i've no ideas

spring raptor
#

(the author bevy_replicon_snap)

scarlet imp
#

I believe I'm having the same issue. Running the simple_box example, the client's box moves significantly (maybe 4x) faster than the server's box. The positions are correcly in sync, it's just the velocity of the boxes that are different.

granite hill
scarlet imp
spring raptor
scarlet imp
#

bevy_replicon

spring raptor
#

But I suspect that both should have the same problem.

#

Since we use delta time on server.

#

I assume that Windows slowdown background apps. And it results into different deltas when you run client and server because only deltas from server being used.

#

I.e. you won't have this problem on real server.

#

Set both fields to UpdateMode::Continuous.

scarlet imp
#

Yep, that fixed the issue. Thanks for looking into it!

spring raptor
scarlet imp
spring raptor
runic nest
spring raptor
runic nest
#

@granite hill I created a PR that adds WinitSettings to the examples. It fixes the problem

timber grove
#

did this just start happening with bevy 0.13? it seems like my project in 0.12 doesn't suffer from this but the new one I'm working on in 0.13 seems to have this happen to it

#

easy fix, but I'm just curious

tardy knoll
tardy knoll
#

Shatur, any plan to add Steam support in bevy_replicon_renet? Or anything that you are waiting for that's missing?

Sort of hacked support on top of it based on the renet example and it works for me. Not sure how easy it would be to contribute this to your crate though. Not sure if it makes sense to support it in any way either tho.

image.png
timber grove
#

So I'm having trouble with stuttery movement on clients (right side of video is client, left side is host). This happens even when running client and server from same machine, same network. It's particularly noticeable when the player jumps or velocity changes drastically. I kind of understand why it happens (player is being simulated on client, then simulated on server, then gets replicated data from server, and applies it to client which then conflicts with client's calculated player position). But I can't think of a good way to implement something that would fix it with replicon.
I'm pretty new to networking in general so maybe there's some best practises for reducing stuttery movement in games in general that I'm unaware of? If anyone has any advice it would be greatly appreciated

spring raptor
spring raptor
spring raptor
spring raptor
# timber grove

If you working on a fast-paced game, you need prediction and interpolation. Replicon only provides core library for replication.
Consider using bevy_replicon_snap to get smooth movement. Or bevy_timewarp.

timber grove
timber grove
#

oh nevermind, I found the github repo

spring raptor
spring raptor
# tardy knoll Any strategy for only inserting this for the current player on clients? I resort...

One more thing!
If you using a different transport, you can't use client.id() from RepliconClient. It's because Renet doesn't export it in a transport-independent way. But you can use relevant transport to get the id. I already notified the author about it (https://github.com/lucaspoffo/renet/issues/153).
Relevant line in the code: https://github.com/projectharmonia/bevy_replicon/blob/4d554de7fc261fea39c301d228b63a15be85e8f5/bevy_replicon_renet/src/lib.rs#L210

tardy knoll
runic nest
#

I insert LocalPlayerId when getting the ClientId for the first time

echo lion
#

Maybe we could emit events for all replication behavior so users can handle them in O(N) instead of O(N + M) (for N replicon events vs M entities to check).

spring raptor
# echo lion <@243426730851696640> what do you think about the perf issue of blueprints due t...

Aware of it. I wish we have Added on archetype level, it would help not only for blueprints, but also for detecting changes for replication... I remember talking to Alice about it, she was quite interested in this.
So I would prefer to solve it on Bevy side, especially since it's not critical.

Events just a little less ergonomic. And you will need to emit them on server when you spawn an entity somehow to support listen servers.

echo lion
spring raptor
echo lion
dire aurora
spring raptor
dire aurora
#

The Without thing ofc only works if your blueprint marker doesn't hold a value, if it does you'd need to check for Changed<BlueprintMarker> to update the other components

spring raptor
#

@echo lion we had an interesting discussion with @dire aurora today: #networking message
TL;DR: Make archetype cache publicly accessible and allow disabling it update.

This way bevy_bundlication could be implemented on top of bevy_replicon.
Third-party crates like bevy_replicon_snap or possibly future rollback crate from @dire aurora won't need to define their own extension traits to register components, they will be able to replace callbacks on the archetypes.

dire aurora
#

Could even just have the implementations (like bevy_bundlication's bundle approach, or replicon's default approach) take some generic to easily inject custom behavior, of course with their default plugin putting in a default "write directly to the component" approach

#

There's also a third person making a replicon/bundlication-like crate (besides lightyear which is way bigger in scope so it probably wouldn't be able to build on top of replicon), and I'd imagine their approach would work with this pattern too

spring raptor
dire aurora
#

Yea, ideally that "combine if needed" step would be easier. Picking serialization functions could be left to the "replication abstraction" (so replicon's .replicate<C>, or bundlication's .register_bundle<M, B, CHANNEL> (that channel isn't even actually used right now 😅)), while a rollback crate might want to modify what the data is actually written to (but it would still be based on archetype, since only some archetypes have NetworkHistory<T> or Remote<T> or whatever)

#

You end up with the weird scenario too where maybe some archetypes need to do the rollback thing, but others need to do some interpolation thing, so now the user has to write their own function that picks which of the approaches to use

#

Or maybe we can register them as a list of callbacks, and the plugin order decide which goes first, plugins could return Option<WriteFunction> with none meaning the default, or whatever the next layer wants 🤔

spring raptor
#

Yeah, ser/de is a separate thing and list of callbacks to intercept writes for third-party crates.
List of callbacks even nicer because it will be empty for replicon and nothing will be called by default (better then having some dummy function).

dire aurora
#

The callbacks would need to be called at the archetype level, and set a function tho ... But it would allow you to just register an optimal version of the function (which is just deserialize the component directly onto the entity)

#

The callbacks could also be optimized a bit because:

  • You can check if a callback will apply for a given archetype at all first
  • Then check per component what function needs to be used there
spring raptor
#

Ah, you suggesting to put both deserialization and "writing into the world" into the same callback list?

I just think that they will have different signatures. Deserialization will need bytes and will create the actual type.
Writing interception will accept the deserialized type and does its own thing.

dire aurora
#

They could have different signatures, one would just call the other, possibly creating some inlined versions for built-in behaviors

spring raptor
#

Got it. So for each archetype user will be able to set custom ser/de function and a list of writing interceptions.

dire aurora
#

You would only pick 1 writing interception I think, it doesn't really make sense to both interpolate and predict the same component

#

But for each component in an archetype you would check a list of possible interceptions

#

And of course if a user had some weird edge case where both are necessary that could be another interception function in the list 😂

#

Clearly we should've been making bevy_architypelication from day 1, good flexibility without hurting performance thonk

spring raptor
#

Sounds reasonable!
I will try to make a draft to evaluate soon.

dire aurora
#

There's possibly also the case of spawning new components to consider ... Tho it's also possible to just call that out of scope cause the plugin can just register a system to add their own components on after reading stuff, then that creates a new archetype which fixes updating for the next tick 🤔

#

Those systems could of course be archetype based too

spring raptor
#

Do you have a marker for your blueprints? If yes, can you insert additional components with the marker deserialization?

dire aurora
#

I was talking more about things like "Transform was added, so now we need Remote<Transform>"

spring raptor
#

Maybe include it into the bundle or you think it will be tedious to add every time?

dire aurora
#

Ideally it would be in the plugin's control, but atm it doesn't really matter since we have no way to batch these inserts

#

If we did have a way to batch them it would be faster if we added it as (Transform, Remeote<Transform>, Velocity, Remote<Velocity>, PlayerId, PlayerName) vs two batches, where the two Remote<T>s get added later

spring raptor
dire aurora
#

Yea, and all those intermediate archetypes make all our fancy archetype-based tricks slower too ferris_sob

spring raptor
#

True :(
But luckily not by much, so I would consider it as a worthy price for the modularity.

dire aurora
#

Right now we can't batch it in any way anyway afaict, so it's probably an optimization for later when it's actually an optimization

#

It would be nice if I could batch inserts when I have code like this 😂

let mut entities: Vec<Entity>;
for entry in cache.list.iter_mut() {
    let archetype = world.archetypes().get(entry.archetype).unwrap();
    entities = archetype.entities().iter().map(|e| e.id()).collect();

    for entity in entities {
        let mut entity = world.entity_mut(entity);
        for f in entry.insert_history_fns.iter() {
            (f)(&mut entity, frames);
        }
    }
}
spring raptor
dire aurora
#

Yea I'll see if anyone in the ecs channel knows of some cursed ways to achieve this 🤔

spring raptor
#

Or not...

#

No, I was wrong, it's not for reflection. It was introduced for scripting some time ago.
I remember that I explored this approach when moved away from reflection, but decided to go with the your type erasure pattern.

dire aurora
#

I think this should work for batching at least ... I'll test it ... Might be useful for replicon too, since it probably has the same issue as bundlication where inserting a few components is insanely slow compared to updating the same set of components

#

In bundlication I actually just run a different spawn function if the bundle doesn't exist 😂

#

It creates the whole bundle at once so it's much faster

spring raptor
echo lion
#

Opening up the API to allow more flexibility is fine with me, as long as the use-case is clear and anticipated (not half-baked).

dire aurora
spring raptor
dire aurora
#

Yea, pretty much ... I think we should first make a version without this optimization tho, better to test the concept first before adding unsafe 😂

spring raptor
dire aurora
spring raptor
#

Yeah, makes sense, no batching.

raw idol
#

if there are no channels configured, can replicon:

  • send data?
  • receive data?
spring raptor
spring raptor
spring raptor
#

@echo lion @dire aurora Opened: https://github.com/projectharmonia/bevy_replicon/pull/221
I need your feedback. I provided API explanation in the description in the second section.
This PR should allow implementing bevy_bundlication on top of replication.
It doesn't contain the discussed separation between serialization and write interception, will be in a follow-up PR to make the review simple.

GitHub
Add control which archetypes are replicated and how by Shatur · Pul...

Changes description
I decided to put a short high-level explanations about what I changed in the PR and why.
ReplicationRules separation
I spitted ReplicationRules into 2 structs:

ReplicationFns. ...

dire aurora
#

Looks fine ... What about component-specific rules like resend times (idk if that's a feature yet, but it's surely an optimization someone might add at some point), would that be part of ReplicationFns?

#

Remove component is also probably less tied to the specific function used, and more to the actual component in question. Tho maybe a function to modify the component removal logic would make sense, like say with rollback I'll need to use a feature that should get unblocked by this PR https://github.com/bevyengine/bevy/pull/11475

spring raptor
spring raptor
# dire aurora Remove component is also probably less tied to the specific function used, and m...

I introduced removal function because it was asked by @RJ: https://github.com/projectharmonia/bevy_replicon/pull/68
I think he uses the same removal function for all components. Do you think that it worth to register it per-component, not per functions?

GitHub
Support custom ticks from FixedUpdate games by RJ · Pull Request #6...

DRAFT - I got my game mostly behaving well (except predicted spawns) using bevy_replicon with these changes.
I will write some tests if you would like to merge this, or something similar. I'm f...

dire aurora
# spring raptor I introduced removal function because it was asked by `@RJ`: https://github.com/...

The thing with the removal is that it might make sense to use a per-archetype removal function (based on the current archetype on the receiving side), but basing it on how it was networked (like you'd need to do with deserializing) would be weird ... Writing and removing components could react to things in the archetype that the server doesn't even know about after all ... Like rollback histories, a Remote<T>, or anything else really 🤔

#

When sending a packet, the data for a component needs to have the id for the serialization function, so the other side can deserialize it. But for removals it could just be a component id

spring raptor
# dire aurora When sending a packet, the data for a component needs to have the id for the ser...

Good point. Removals is a different abstraction level, it's more like the planning write interceptions, something that prediction libraries do.

I will move it out from ComponentFns.
Also, you mean not ComponentId itself, but some separate ID for component removal functions, right? Because ComponentId isn't portable, it depends on components access order.

Feel free to suggest things like this, even naming, it's always welcome.

dire aurora
#

Yea, not ComponentId, before I guess things were just networked based on the component they are, and now it's based on the function used. So in a way the removals use the "old system"

echo lion
#

@spring raptor maybe you should put your PRs in draft for a couple days first lol. You always end up pushing a bunch of commits if I wait long enough. It's fine and good, it increases the PR quality. However, reviewing in the middle of changes doesn't work.

spring raptor
echo lion
#

I am the same way as you, I like to push the first draft that feels good, but then end up adding a bunch of trailing commits :)

spring raptor
#

@dire aurora just suggested a few adjustments about removals, I applied them. Then I realized that for custom rules removals and despawns should be handled separately, opened the API for it too.
Luckily we have all this already abstracted, I mostly opened the API and refactored structs a bit.
Let's wait for the confirmation about the API from @dire aurora and I think it will be ready for review.

echo lion
#

Btw I am pretty sure I will fork renet to add my netcode protocol changes (all my open PRs: https://github.com/lucaspoffo/renet/pulls). Anything I should consider before diving in? Part of me thinks I should also fork the netcode spec to document my changes: https://github.com/mas-bandwidth/netcode. In addition to those PRs I also want to add a rule allowing fresh connect tokens to usurp old ones, which improves reconnect cycle times for certain edge conditions (clients force-quitting, client crashes, etc.).

spring raptor
echo lion
#

Of course, my bevy_girk_demo is stuck on bevy v0.12 because it depends on bevy_lunex, which I desperately need to replace. So I need to stay focused on UI.

echo lion
spring raptor
echo lion
spring raptor
echo lion
#

It's not maintained and I want to switch to something based off bevy_ui, since I discovered it's not easy to write a custom UI solution. Clipping requires a custom render pipeline. Also, a custom solution won't have access to the bevy_ui feature-set. My pursuit of satisfaction definitely impedes progress in some ways haha.

#

I am really optimistic about this crate though: #devlogs message (purely a widget library on top of bevy_ui). Planning to study the design and see how I can merge it with my current UI ideas.

spring raptor
spring raptor
echo lion
#

Sadly my game requires a ton of UI, so I am really trying to find good patterns. bevy_cobweb_ui was going well until I ran into the wall of custom clipping. Now I want to see if I can adapt the crate to use bevy_ui under the hood, basically a usability layer on top. Usability layers appear to be the long-run solution for Bevy UI, everyone seems to be converging on that.

spring raptor
#

Going to sleep right now, it's 3:45 AM for me. And I have to go to work tomorrow. Lackily I work from home 😄

echo lion
dire aurora
spring raptor
#

@echo lion marked as ready for review.

dire aurora
#

And I don't even replicate 1000 entities yet

spring raptor
dire aurora
#

Pretty sure the renet overhead should be tiny when I'm only sending a few packets to 1 client 🤔

#

I've definitely had issues with renet before tho, when I accidentally replicated thousands of entities with broken change detection 😅

spring raptor
spring raptor
dire aurora
#

I send a string, u32, Vec3, Quat, Vec of enums and some bools in my benchmark ... That's also just the sending part, being new or not makes no difference in my crate ... The receiving side is slower because it spends most of the time on actually inserting the components :')

spring raptor
dire aurora
#

If you just accept one component per entity it wouldn't make a difference, but if you do say 5 at once it makes a pretty big difference

#

The receiving benchmark ran 5x slower before I implemented an ugly check to spawn a whole bundle at once if it's a new entity

spring raptor
#

Ah, right, we definitely need a benchmark with multiple components.

spring raptor
#

@echo lion found a good corner case.
In order to detect if an entity was spawned, we check if it contains marker (Replication component in case of component-based rules).

@dire aurora do you have your own marker per-bundle or you use a global marker?

dire aurora
#

I use my Identifier component to check if an archetype should even be replicated at all

spring raptor
#

Right now I consider Replication component as a marker for component-based rules. And I see a few options:

  1. Consider it as a special marker for all rules instead. Even if @dire aurora uses a separate markers for each bundle, he can just also insert Replication as well. This way code for spawn and despawn detection will be unified. But is see a small problem. What if component is defined by component-based rules and bundle-based rules, for example?
  2. Continue to consider Replication only as a marker for component-based rules. Then we need to provide an additional API for it.
dire aurora
#

Arbitrarily networking things just because they match some registered bundle would probably have horrible side effect 😂

spring raptor
#

Got it! So not per-bundle, a global component as we do.

#

I think it will be better to go with (1), but not sure what to do about the possible problem of rules collision.

dire aurora
#

I can technically define per-bundle markers, but that's really more of a hack, I can add something to the bundle and just never send or spawn it, which basically makes it a required marker

spring raptor
#

I will adjust the PR, it currently coupled with component-based rules.

spring raptor
#

Pushed changes, updated description.
Basically adjusted Replication a little to keep it rule-independent. And as a nice bonus third-party libraries won't need to handle despawns.

spring raptor
#

Working on a test with multiple archetypes since this case is not covered by our code.

spring raptor
#

Done :)

spring raptor
#

I think that I probably need to move component-based rules system into its own Plugin... I will do it a bit later today.

spring raptor
#

@echo lion Okay, now everything should be in place :)
Sorry for the followups, but now rules are isolated better. This helped me realize that we need a system set for removals. From the API side nothing changed.
Each commit is a separate change to made the review easier.

spring raptor
#

Koe suggested some interesting changes to the API.
Going to experiment with them!

spring raptor
#

Experiment is going well!
Going to push it soon, need to polish.

dire aurora
#

Does replicon have any features that allows sending different data to different clients? Like only sending certain components only to the owner of an entity or other clients on their team? And I guess also for sending diffs since last confirmed tick per client 🤔

echo lion
dire aurora
#

I only see things about entity visibility?

echo lion
spring raptor
spring raptor
#

Edited ^

dire aurora
#

Hmm ... So some kind of per-component thing still needs to be added ... Not sure what the ideal API is here, there's a few cases where you might want to send one component partially, and quite a lot of cases where you only need to send certain components to the owner of an entity or allies of a client

#

I have the owner vs others thing in bundlication, at a per bundle level of course, but I never made anything that would work for PvP usecases 🤔

spring raptor
#

Yes, need to figure out the API.
For entity visibility we have a method on ConnectedClients resource (you retrieve a connected client and configure it). Originally I planned to make Rooms API, similar to what naia have. But ended up with this barebone interface because it's a little faster and if you don't need any visibility control at all, it's zero cost. And it's possible to build an abstraction on top of it if needed.

In theory we could have something like this for components too. But maybe we could come up with something nicer.

dire aurora
#

In theory, per archetype we could know "these components with these serialization functions should be sent to everyone, these should only be sent if the client and entity match in X way, and these should only be sent if the client and entity match in Y way" ... Then we could have some kind of visibility for each of these groups

#

And the groups could just be things like "is owner", "is allied", "is enemy", etc

#

If you can use a different serialization function depending on the group I think you can handle all major ways to save bandwidth besides being able to send component-level diffs, which requires a different API anyway

echo lion
#

Needing to send different components to different clients kinda feels like a design smell. At that point you have quite dense structural logic.

dire aurora
#

This is a thing that happens in almost every online game. A player receives way more info about their character than other characters, you get more information about your party members than enemies, you might even get a different amount of data depending on how close entities are

echo lion
echo lion
spring raptor
spring raptor
#

However, the components in Unreal are not the same.
So yes, we need a good API suggestion.

#

I have a bit rough week, working overtime for my company to prepare a demo. Should end on Thursday.

Didn't have time to finish an alternative PR. But it's almost ready.

dire aurora
# echo lion Right, but all that data doesn't need to be slammed onto the same entity, does i...

Even if you do split some data into separate entities, you'd just end up with components linking them you need to replicate ... Splitting them up needlessly would also be counter to saving bandwidth since now there's more entities to network ... And of course splitting entities up just because it networks nicely would be kind of missing the point of an ECS-level replication API, it would become a different more clunky version of "write custom systems to collect the necessary data and send it" you get when using something like renet directly

#

As for performant API I think the one I suggested would be the most perfomant, you'd just specify that X player can see "Owner" components on entity Y ... It could work on visibility stuff to create very little overhead, especially if we use bitmasks for the groups instead of some list of groups ... But I'm not sure if that would be the cleanest API, nor do I know how that would translate to the component registration API

echo lion
#

So visibility on an entity would be tied to your component access level, and then each component could be assigned an access level on the entity.

#

The main perf issue is you need a lookup per-entity per-component to figure out the component access levels. Getting entity/client access levels would not be expensive since it can be inlined to the existing visibility system.

#

I need to stop getting nerd sniped all the time lmao.

dire aurora
spring raptor
#

I think that @dire aurora suggesting to create a logic like this:

  • Is this client an owner? If yes, here is a list of components. No? Here is another list of components.
    Right?
dire aurora
# echo lion The main perf issue is you need a lookup per-entity per-component to figure out ...

If visibility gives us a bitmask of access levels, and we just check that against the access level for that component in the current archetype before actually sending it, that should be a pretty cheap check ... Not sure how this would interact with removal of components, changing access levels is probably one of those annoying edgecases you can't escape ... Inviting someone into your party in an MMO, an entity getting close enough to get some "nearby" access level, that kind of thing

echo lion
spring raptor
#

Ah, yes, that's basically what access levels do 😅

dire aurora
spring raptor
#

What if we do it not per component, but per component group?

dire aurora
#

So keep one list of components per access level? That might work too, not sure if it's more or less efficient than checking the access level on each component 🤔

echo lion
#

Hmm what about a marker component on entities, and in the marker component you specify the access level strategy for components. Then strategies are registered with the replicon central authority. The marker component lets us filter archetypes so only access-constrained entities need extra lookups for mapping access strategies.

#

And the access strategy lookup can be an optimized match on component id, rather than hashmap lookup.

dire aurora
echo lion
echo lion
#

We always need to look up entity visibility in case of blacklist vs whitelist, but component access levels would be different.

#

The risk with globally-assigned access levels is you get stuck needing a component at different access levels, where new-typeing isn't an option.

dire aurora
#

As long as you don't model some really cursed access levels like "Player 1" "Player 2" anyway 😂

#

But you can model that same case as client 1 has Owner & Ally for entity X, only Ally for entity Y, and none for entity Z ...

echo lion
#

The simple model in my mind is an access level can see itself and all lower levels. More intricate access strategies will be more expensive (although whether that's hot-path expense I don't know).

dire aurora
#

The problem with modelling it as actual levels is that you lose the ability to compose them. But with a bitmask you'd retain basically the same cost (well it's a bitor + cmp instead of just a cmp, but that seems extremely minor), only issue is you'd be limited to 64 of these access levels/groups

#

I don't see how anyone would even get over 10 tho thonk

#

I mean maybe if you make them very fine grained and combine a few sets say:

  • "Owner", "Party member", "Allied", "Neutral", "Hostile"
  • "Very close", "Close", "Far away", "Very far away", "Extremely far away"
  • "Friend", "Guild Member", "Alliance member", "Stranger"
spring raptor
#

Yes, this sounds more flexible. Like in a physics engine.

dire aurora
#

Yea pretty much exactly like the physics layer in a physics engine

dire aurora
spring raptor
#

Crunchy days on my works are over, finalizing the PR right now. Maybe I will be able to submit it today.
This is approach is too different, so I did it from scratch.

spring raptor
#

@dire aurora @echo lion I opened https://github.com/projectharmonia/bevy_replicon/pull/224
I would appreciate your thought about the API. I marked the PR as draft because I need to write tests, but the code and documentation is ready. For more detailed description take a look at the docs in replication_rules.rs.
Once I will write tests, I will mark it as ready for full review.

GitHub
Replication groups by Shatur · Pull Request #224 · projectharmonia/...

Replaces #221.
Previously, the API did not efficiently support mixing component and bundle groups. Under the previous architecture, every archetype with the Replication marker matched a component-b...

echo lion
spring raptor
dire aurora
#

This approach would mean you can combine them more easily ... But it also seems a lot more limited, or it would require a ton more features in replicon ... If someone wanted to make an API based on specialization to reduce bandwidth usage (like (Player, Transform) could use a smaller format for Transform than the normal Transform or Velocity), the current code would just send both formats 🤔

echo lion
spring raptor
echo lion
dire aurora
spring raptor
dire aurora
#

Only if it was introduced by your GroupReplication

spring raptor
#

Ah, I think I get what you mean.
But no, it's not an archetype definition. I.e. PlayerBundle will match any archetype with Transform and Player.

#

@dire aurora forgot to ping ^

dire aurora
#

Right now in bundlication I don't actually support this correctly either, but the need for the ability to specialize the format with more specialized bundles (groups in this case) is painfully obvious. Since that way you can prototype things easily but still skip irrelevant components in larger bundles (For example: Most character controllers don't need angular velocity and full Quat rotation, but regular dynamic bodies most likely do)

spring raptor
dire aurora
#

That check just makes it panic if you try to do this ... If we wanted to support this we'd need some priority value, and some logic so it overwrites components with lower priority rather than trying to add the component twice

spring raptor
#

You mean that you need rules that are subset of other rules?

#

I.e. support replicate::<Transform> and replicate_group::<PlayerBundle> with PlayerBundle contains Transform and other components?

dire aurora
#

Yea, and in this case PlayerBundle's Transform would have to overwrite Transform

spring raptor
#

🤔

#

But can't it be solved with markers? It's just cheaper to do.
Like for Player you replicate Transform in one way and for StaticObject you replicate Transform in other way?

dire aurora
#

Yea, except StaticObject doesn't have any special replicon rules because you don't consider if necessary to optimize that

spring raptor
#

Ah, you mean that it would be more convenient to mark a component for replication once and provide an optimization for one specific case?

#

It's possible to support it, but it will have performance impact on archetype caching and removals...
The same problem would be with the previous archetype-based solution. Except handling all this cases would be on a third party crate 😅

dire aurora
#

The archetype part shouldn't be super heavy, you could give a priority to each component, and if a new rule has the same component as you already have in the archetype, you can discard the new one or overwrite the old one depending on which one is higher

#

But yea removals would be a lot more difficult than if overlap isn't allowed at all

spring raptor
#

And priority is determined analyzing adding rule? Like if a rule is a superset of another rule, you increase the priority of the added rule. If a rule is a subset of another rule, you increase the priority of its superset.

spring raptor
dire aurora
#

And you might need a custom case for the "don't serialize this" 🤔

spring raptor
#

Yes, should be better then just panic.

spring raptor
dire aurora
#

Yea, like PlayerBundle might not need AngularVelocity, but that component is registered as a stand-alone component ... But the desired behavior is for PlayerBundle to take priority

spring raptor
dire aurora
#

I think manual should be fine, or maybe we can have the constructor default it to the number of components, but add a method to override that when necessary 🤔

spring raptor
#

I like it!

#

Thanks, will work on it tomorrow.
Going to sleep right now 💤

spring raptor
dire aurora
#

Yea, removals basically have two cases:

  • The group had no overlap, so if any of the components disappear you can remove all of it
  • The group has overlap, so if any of the components disappear, only the ones that are not in other groups can be removed
    The first one is easy but the second one seems harder ... I'm also not entirely sure what the removal behavior should actually be here, if only Player was removed, removing Player and Transform on the other side because that's what was in the bundle does feel a bit odd 🤔
#

This is why I never built removal support for bundlication 😂

spring raptor
#

My current thoughts:

  • Split component field with ComponentId and SerdeFns into two Vecs and sort first vec with ComponentId. It will allow us to quickly check if one set is a superset of another (Vec::starts_with).
  • When doing removals, collect matched rules first and then remove subsets.
dire aurora
#

I don't think starts_with gives you accurate results there ... We could just move the expensive checks to the finish part, and keep the processed info tho 🤔

spring raptor
# dire aurora I don't think starts_with gives you accurate results there ... We could just mov...

I don't think starts_with gives you accurate results there
But why? I would expect [A, B].starts_with([A]) return true.

We could just move the expensive checks to the finish part
Yes, I thought about it too.
I was thinking about calculating rule subsets and sort rules itself by priority. And on iteration ignore subsets if a rule matches. But it will require allocation and a lookup for each other rule...

dire aurora
#

When you sort it by ComponendId it might become [B, A]

#

This gets even worse when sets are more than one item because another item could be placed in between

spring raptor
#

Ah, right :(

dire aurora
spring raptor
#

Maybe it's not that bad. I don't think an entity will have a lot of matching rules, so it won't be a lot of lookups.

dire aurora
#

You check it per archetype, not per entity, so the number of iterations is relatively low

spring raptor
#

Right, I wanted to write per archetype :)

#

Removals will be per-entity, unfortunately.

#

But it's not that bad, I don't expect people to remove a lot of components

dire aurora
#

For removal in theory in the finish step a list could be generated for each group's components of which components would need to be absent before it can be removed, that way the cost at removal is just checking for a few components, which should be a very tiny cost compared to how expensive the archetype move from that removal most likely was

#

Tho it might need to be a combination of components ... Like "Transform can be removed if the entity has neither [A, B] nor [C]"

#

Or alternatively we could just check what components got replicated in the previous archetype vs what gets replicated in the current archetype, and remove everything that's missing

spring raptor
#

I wondering what we need to do if there are two rules like this:

  • A
  • (A, B)
    You remove B, what happens?
dire aurora
#

If you remove B it should only remove B of course, since A will switch to the other rule and still be replicated

spring raptor
#

Sure, but it means that we probably should make removals per-component instead of per group...

dire aurora
#

Yea, you can't really do any useful per-group removals unless they can all get removed at once

dire aurora
spring raptor
#

In the current implementation I just warn users that they can't remove groups partially and require them to be disjoint.
It "solves" all these problems, but not very ergonomic.

#

But we probably should switch back to per-component removals to allow this case.

#

specialization case*

dire aurora
#

per-component could work ... Maybe it could combine things if and only if all components in a group get removed ... Not sure if that adds up to any meaningful bandwidth reduction since I have no clue how often removals would be done

spring raptor
#

Yeah, I would expect them to be quite rare.

dire aurora
#

I guess the main usecase is if people make components for things like Dead or Poisoned, that can be reversed?

spring raptor
#

Yeah, I imagine it something like this. And it's usually a single component anyway.

#

One more thing about removals. Let's consider an entity with (A, B) rule. You remove only A. What happens?

dire aurora
#

Well that's the one I'm not sure about ... I feel like B should also be removed, but it still exists on the server so that also seems kinda wrong?

spring raptor
#

If there is a rule for B, that it's obvious. But what if it's not, B should also be removed?..

#

Yeah...

#

And the hardest part if how to detect it. This way we not only need to check for removals, but also for all existing components and try to determine which rule is removed.

#

Maybe consider it as "not allowed", like overlaps?

dire aurora
#

If we somehow know that it's happening we could ignore B but throw a warning for it ... If we don't easily know I guess we can just ignore B for now ... The (A, B) bundle is totally valid as long as you don't remove only A 😅

spring raptor
#

Thanks, then I will replace the current panic in finish with the discussed pre-calculations and update how archetypes and removals are handled.
In replicate_group I already have a warning rule about overlaps on the same entity and removals. Will just remove the warning about subsets.
I wish we could remove all warning and make it just work, but I guess it's not a big deal. Subsets was the most concerning, glad that we found a way.

#

And maybe warn about overlaps with debug_assertion enabled.

dire aurora
spring raptor
#

Pushed the changes. Current tests pass, but I need to write tests for groups, working on it now.

spring raptor
#

Marked as ready for review! Managed to handle overlaps too, no warning needed, will work as you expect: no double replication of the same component.

Don't get scared by the big diff, it mostly tests because of many possible cases. Logic itself are not complex. Except maybe removals are a bit tricky.

spring raptor
echo lion
#

I will start reviewing #224

echo lion
#

@spring raptor what do you think about changing replicate_group<Pattern> to replicate_group<Pattern, Additional>? Pattern is checked first and if it matches then Additional are checked. That way you can add types to Additional that are normally individually-replicated. (just spit-balling, haven't thought deeply yet about how this would be used)

spring raptor
echo lion
spring raptor
echo lion
spring raptor
crisp stump
#

beautiful

dire aurora
# spring raptor Marked as ready for review! Managed to handle overlaps too, no warning needed, w...

Haven't looked in-depth yet but the tests and archetype iteration logic look correct ... No priority for the rules yet (I guess also as a follow up PR, like filters?), which does create the weird edge case where if rules overlap now it feels random which one gets picked ... Like koe said, a warning message could be good there, unless the serialize function is identical or once we get priority if the priority isn't the same

spring raptor
dire aurora
#

Oh, I see now, the archetypes gets sorted by priority ... I guess we're just missing tests to make sure priority is respected when adding the components to an archetype

spring raptor
spring raptor
dire aurora
#

Oh yea I meant rules, sorting archetypes wouldn't do anything ofc

spring raptor
spring raptor
#

I went ahead and pushed a small commit.
Let me know if you have a better idea.

echo lion
spring raptor
#

Awesome, thank you!

echo lion
spring raptor
#

Thanks a lot, adressed everything.
@dire aurora fine with you too, good for merge?
We will be able adjust the API additionally if you find any inconveniences. Hard to write a good interface from the fist try, but I tried by best :)

dire aurora
#

The changes look fine ... I can probably try to make bundlication's API on top of replicon soon-ish, will need a way to set a write function if I want to test it propperly in my game however (Cause I'll need to write to histories for rollback)

spring raptor
dire aurora
#

If we can get bundlication's API to work in replicon we just need to make lightyear use replicon for ECS-level replication, then we can finally stop the networking fragmentation 😂

spring raptor
#

😅

#

BTW, feel free to left reviews if you spot something, it's always welcome.

spring raptor
dire aurora
#

Would also be interesting to see how bad the refactor would be ... I'd imagine most of the changes would be related to entity mapping

#

Since obviously replicon uses entity mapping and not the weird Identifier thing bundlication has 😂

spring raptor
dire aurora
#

Not quite, but it's pretty easy to change in all but 2 cases

#

Those 2 cases being actual pre-spawn cases, and I don't even really handle them all that well right now either, so I'll probably need to redesign that workflow anyway

#

It should still easily beat the renet -> bundlication refactor ... Cause that was 88 files changed, 3126 insertions(+), 4159 deletions(-) 🙃

spring raptor
spring raptor
dire aurora
#

Replacing

commands.spawn_with_id(
    IdentifierType::Creature,
    counter,
    Bundle,
);

with

commands.spawn(
    Replicated,
    Bundle,
);

and removing all the counters and IdentifierType stuff could add up to a fair few lines tho, especially since the formatting would change for really long chains 😂

spring raptor
# dire aurora Replacing ```rust commands.spawn_with_id( IdentifierType::Creature, coun...

But at least changes aren't that hard, I imagine that from renet -> bundlication you had to rewrite some logic?

I noticed you used Replicated instead of Replication. I remember that @echo lion suggested some time ago to rename into Replicated but we decided to postpone it because we already had a lot of changes. Right now we don't have a lot of breaking changes for users, so maybe worth to rename...

dire aurora
#

Oh, yea Replicated feels more intuitive than Replication but I can see how that would be an annoying breaking change ... You could reexport it for one version to "deprecate" it, but I'm not sure if you can give deprecation warnings for type aliases 🤔

spring raptor
#

Yeah, I will experiment with it

dire aurora
#

It could even be Replicate, I'm not sure if there's a strong naming convention for components tbh 😂

spring raptor
#

Yeah, but the current one is definitely no good.

spring raptor
dire aurora
#

Bevy is also fairly inconsistent, bevy_render has NotShadowCaster/Receiver, but also NoFrustumCulling 😂

spring raptor
#

Agree 😄

dire aurora
#

If it's Replicated and people think Replicate, chances are auto complete will have exactly 1 suggestion too 😂

echo lion
sharp roost
spring raptor
echo lion
#

Another big PR done, wooh 🎉

spring raptor
#

Thank you a lot for your thorough reviews!

spring raptor
spring raptor
#

@dire aurora working on deserialization and writing separation. Here is how I see it:

  • Pass writing function into deserialization function (because we can't return OwningPtr pointing to an object on its stack)
  • Convert actual type into OwningPtr via some cursed stuff.
  • Pass it into writing function.

Maybe you have a better idea in mind? Because the approach is a bit ugly...

dire aurora
#

The writing function wouldn't have much use for an OwningPtr, and besides inserting new components there's no real benefit to using it

#

The writing function I'd need for rollback would need to write the value to a VecDeque, based on which tick it was from (or discarded if it's too old)

spring raptor
spring raptor
dire aurora
#

Hmmmm ... It's a difficult one ... You kind of have two usecases too ... One where you want a new value from the deserialize function and one where you want the deserialize function to write to an &mut C

#

I have that in bundlication too, tho there it's to reduce the noticable performance hit you get when constantly creating new alloc types with deserialization functions

spring raptor
dire aurora
#

In bundlication I have an (optional) function in my trait to deserialize something to an &mut C instead of returning a new C. It's automatically used when the component exists ... When you don't specify it it just defaults to using the function that returns a new version

#

If writing functions got to choose which version to use it could then be a lot less hacky

spring raptor
# dire aurora In bundlication I have an (optional) function in my trait to deserialize somethi...

Ah, read_in_place, I remembered!

We could allow to have two functions, but I afraid that it won't solve the problem.
Right now deserialize can do anything: you have EntityMut inside and you can insert a new C or update it or insert into something like Predicted<C>. But the problem is that deserialization is coupled with writing and it's not quite convenient to write component in a custom way and use a prediction crate because both crate and user want to overwrite the same function :(

#

I just imagine that in bundlication you want to specialize components and have a prediction as a separate crate.

#

In theory you could chain them inside bundlication.

#

Because you know the type at the time you generate your function.

dire aurora
#

If the writing function calls whichever version of deserialize it wants, you should always be able to write a write function in a non-hacky way

#

In bundlication the read_new function is only used for new components, but these write functions can of course look very different, if you write to a VecDeque you might always need a new value, but a Remote<C> would always want to deserialize to an &mut C (well it would be an untyped Ptr probably)

spring raptor
# dire aurora In bundlication the read_new function is only used for new components, but these...

Sorry, I don't think I get it. Could you elaborate?

Just for the context, function currently looks like this: https://github.com/projectharmonia/bevy_replicon/blob/70a3bdc60d42c805c78cb400c8508a9dde5d5e94/src/core/replication_fns.rs#L137
And user or a crate can replace it to write into Predicted or serialize it differently, but not both automatically. It's possible to manually for user to specialize deserialzation and write into Predicted, but it needs to be done manually.
Bundlication will be able to automate this kind of chaining.
We wanted to try to separate writing and deserialization, it would be more convenient. But looks like chaining without knowing a type would be a little hacky. So maybe keep it as is? Or want to suggest something else?

#

Fixed typo: serialization -> deserialization.

dire aurora
#

Rather than have the deserialize function take a write function, you can flip it around. If the write function can either pass a valid pointer to deserialize, or get an OwningPtr and immediately cast it back to C not a lot of ugly unsafe stuff needs to be done, and it should still be possible to get some optimizations like read_in_place, or eventually per-component diffs (which kind of depends on having support from both the write and deserialization functions anyway)

#

If deserialize always creates an OwningPtr, and passes that to a write function you wouldn't be able to get any of those optimizations in later

spring raptor
dire aurora
#

Hmmm ... You kind of can but you also can't ... If we make methods to deserialize to &mut C or return C we should be able to work around the issue 🤔

#

The code I have that uses OwningPtr just gets a reference to a correctly sized space in memory, and it writes there ... That way it will stay in scope until it's no longer necessary (bevy mem copies the component into the ECS, so it can be dropped after it's written)

#

If the write function is typed (and it needs to be, because it can't query things like Remote<C> if it doesn't know what C is), it could provide the type as a generic argument those menthods, and the ugly Ptr/OwningPtr mess can be contained to only deserialize

#

Could even add an extra check (maybe only with debug_assertions enabled) to make sure the C argument passed to deserialize actually matches the TypeId for that deserialize function ... Then calling those methods could be considered safe

spring raptor
dire aurora
#

No you don't write to the component, just any &mut C ... It could be a pointer to an element in an array, or the contents of Remote

spring raptor
#

Ah, I get it

dire aurora
#

But you obviously still need to do something with C, because that's the type you get from the deserialize function

spring raptor
spring raptor
#

Ah, you probably mean to use OwningPtr::make inside the deserialize function and pass closure into it from writing function. Makes sense to me.
Let me play with this, will open a prototype PR soon (going to sleep right now, so tomorrow after work).

dire aurora
#

You can't use make iirc, since the value will get dropped out of scope

#

Default could work, but maybe leaving the space uinitialized could also work 🤔

#

Actually I guess if closures are involved it would still be in-scope but that would add an extra function call and thus be slower

spring raptor
spring raptor
#

Maybe we can "pipe" on registration stage?
Deserialize function will return C and write function will accept C. And on registration we accept a single "piped" version of both functions.
For custom deserialization user can write its own function and pipe it with the default writing or with a "predicted" writing provided by a third-party crate.
This way it should be inlined and completely safe.
What do you think?

dire aurora
#

Problem is we can't reasonably decide what write function should be used at registration ... Since only an archetype with Predicted and a prediction history would need the predicted write function, otherwise you can't have such write functions coexist, which would be pretty bad. The obvious example would be a game where you don't predict the whole world: You predict the client and maybe a few skill effects they caused, but interpolate every other entity. That's two write functions, and possibly the default is also used for things that don't need to get interpolated 🤔

spring raptor
dire aurora
#

That would mean that if we had say 3 of these systems we'd need to make a custom one, which is fine, but also check multiple components for every single write we do

#

And if it's like the only client is predicted thing, that means we do these extra checks so only 1 entity can ever match

spring raptor
dire aurora
#

You don't predict just a specific component, you'd have to add it to every networked component the player can have

spring raptor
dire aurora
#

If we pick a write function based on the archetype we could bypass that problem

#

Every write function registered for a component could have some required component ids, if they are present it could be used ... Then we just pick the one with the highest priority if there's a conflict, or use default if we have nothing

spring raptor
dire aurora
#

Hmmm ... Maybe we could do some other check to quickly know which function to call, and make it so that it only needs to get checked once per entity update instead of per-component

spring raptor
#

Yeah, thinking about the same...

dire aurora
#

I think interpolation and prediction should always have a marker to look for, after that the major differences could be handled by inserting a new component vs updating a component maybe? Since we kinda want to separate those two anyway so we can optimize spawning of new entities

spring raptor
#

One more question. When you receive components, do you care about all intermediate values?
For example, you can receive multiple updates in the same tick. Right now I discard old ones. Is it acceptable for prediction?

#

I assume that it's okay, but I decided to clarify.

dire aurora
#

For prediction I'd need all the updates

#

But for interpolation you only need the newest one

spring raptor
#

I see, so we need a special case for predicted entity anyway.

dire aurora
#

Yea we might need the ability to specify if write functions need to get old values

#

You only need the history if you do full rollback, if you only predict 1 entity it's already not needed anymore, and interpolation obviously doesn't care about it at all

spring raptor
#

Got it. So we need some sort of switch between "full history" / "only last value" and the ability to override writing function per-entity. Let me think about it a little more...
OwningPtr or "piping" won't allow us to overwrite writing function per entity, only per component which isn't nice.

spring raptor
#

@dire aurora What if we use recently added specialization for groups for selecting writing function per entity?
I.e. in prediction crate you introduce a special marker and register a special writing function for combination with this component?

As for switch between history mode, it could be a marker component on replicon side.

dire aurora
#

I think the switch to history mode would need to be based on the write functions, because my prediction function might want history, but I'm pretty sure interpolation wouldn't suddenly want old values because there's 2 predicted components on that entity 🤔

#

The markers are already kind of there anyway I'd imagine

#

For my rollback code at least I have a Predicted marker component

spring raptor
dire aurora
#

We could have the registered write functions also register if they want the last value or full history

spring raptor
spring raptor
#

@dire aurora how about this? https://github.com/projectharmonia/bevy_replicon/pull/227
I played with it and I think OwningPtr is more elegant then "piping".
I also managed to use OwningPtr::make with only 2 calls instead of 3. Well, technically there are 3 calls, but one of them will be inlined. So it should be on par with writing into &mut C, but safer. Partial component ser/de and writing in place are still possible: deserialize and write should operate on the same type, but it doesn't have to be C. I.e. it's possible to use (Option<FieldA>, Option<FieldB>)

Most of the changes are documentation adjustments.

GitHub
Split writing logic from `deserialize` by Shatur · Pull Request #22...

Split writing logic from deserialize into separate write function to customize it independently.

#

History mode will be a follow-up, this one includes only separation.

dire aurora
#

OwningPtr::make can't be on-par with writing to &mut C, if the type is an alloc type it requires a new type to be created

spring raptor
spring raptor
#

@dire aurora Trying the following:

#[allow(clippy::uninit_assumed_init)] // Component will be initialized after deserialize.
let mut component = MaybeUninit::<C>::uninit().assume_init();

(deserialize)(entity, PtrMut::from(&mut component), cursor, entity_map)?;

entity.insert(component);

Ok(())

I'm a former C++ programmer and the following would be okay in C++, but is it fine with Rust too?
Because a test with a huge Vec as component says otherwise 😅 Results in double free or corruption (out).

#

It was writing fn. Deserialize looks like this:

    let component: C = DefaultOptions::new().deserialize_from(cursor)?;
    *ptr.deref_mut() = component;
#

Yeah, looks this the following could crash:

let mut component = MaybeUninit::<Vec<u8>>::uninit().assume_init();
let ptr = PtrMut::from(&mut component);
*ptr.deref_mut() = vec![0; 1200];

If uninitialized Vec points to garbage, it will try to free the memory.

#

Fixed typo in the code above ^

dire aurora
#

Hmmm, I wonder if Vec stores some non-zero state as default ...

spring raptor
#

Yes, it does.

#

std::ptr::Unique<T> should be non-zero.

dire aurora
#

I guess the best we can do would be pass in two versions then, and have it default like bundlication 🤔

spring raptor
#

Will we benefit from in-place deserialization in this case?
We can't use it for Vec or String.

dire aurora
#

If we can generate a non-default impl by default we would

spring raptor
#

I assume that the use case was to deserialize bytes directly into alloc fields, but without it, how it can help?

dire aurora
#

You want both implementations so write can use whatever it needs

spring raptor
#

Not by default. But how user can override it and benefit from it?

dire aurora
#

No, you'd need deserialize_new and deserialize_in_place, and just one write

spring raptor
#

Ah, right, sorry, I meant this.

#

How deserialize_in_place can be used by user to avoid extra alloc?

dire aurora
#

The user doesn't use it. If the write function has an existing value, like when you have Remote<C>, you can use deserialize_in_place

#

If people specify a custom deserialize function it can be optimized further of course

#

But the in_place version should probably be optional cause it's annoying to specify everywhere

spring raptor
#

Ah, I think I get it now! Let me try...

#

There is another problem 😅
In order to update a component from an entity in-place, I need to borrow it mutably. But I also need to borrow world mutably in order to spawn new entities while mapping components.
Let me think if I can avoid it...

spring raptor
spring raptor
dire aurora
#

Hmmm, it doesn't really avoid creating a new C yet, but at least this API would be the right workflow for adding in-place later without breaking stuff too much 🤔

#

I'm also not sure if write should be used for both insert and updating, but I guess that kind of depends on if replicon already checks if the component exists or not 🤔

#

Actually wait we'd want to bundle inserts later so we kind of have to do that ofc 🤔

dire aurora
#

But cart also said something about making spawning entities from anywhere safe

spring raptor
#

Hmmm, it doesn't really avoid creating a new C yet, but at least this API would be the right workflow for adding in-place later without breaking stuff too much 🤔

Yep! Default implementation basically always creates C, but user can override it.
Or you mean that updating C when possible is faster then inserting?

spring raptor
dire aurora
spring raptor
#

Ah, there is even World::get::<C>.

dire aurora
#

We could also say that write and insert are separate functions, and not pass EntityWorldMut to write, and only EntityMut, then we can safely also pass in Entities (and whatever selse is needed for mapping entities)

dire aurora
#

I wonder if it's possible to just transmute the function pointers back to safe types ... Something like this:

struct DeserializeFns {
    new: unsafe fn(),
    in_place: unsafe fn(),
}

impl DeserializeFns {
    pub fn deserialize_new<C>(&self, cursor: &mut Cursor) -> C {
        let new_fn: fn(&mut Cursor) -> C = unsafe {transmute(self.new)};
        (new_fn)(cursor)
    }

    pub fn deserialize_in_place<C>(&self, dst: &mut C, cursor: &mut Cursor) {
        let in_place_fn: fn(&mut C, &mut Cursor) = unsafe {transmute(self.in_place)};
        (in_place_fn)(dst, cursor)
    }
}
spring raptor
# dire aurora I'm also not sure if write should be used for both insert and updating, but I gu...

Right now we don't check if component is present.
Splitting write into insert and update will also require splitting deserialization too.
But since we don't check if a component is present (at least for now), maybe it's better to have a single write and insert that can be specialized to perform in place writing if needed? We can't provide a default in place that actually helps anyway, only generate it with macro (maybe a task for bundlication?).

spring raptor
dire aurora
#

Replicon would pass DeserializeFns to write, and because write is typed it can give C to the methods

dire aurora
# spring raptor Right now we don't check if component is present. Splitting `write` into `insert...

Just like how there's a deserialize_component, you could have a deserialize_component_in_place that takes the &mut C. It would allow reusing alloc types, but any further optimization would require custom logic by the user. For example, I have a few components that use in_place to keep an older tick value if the new value is exactly 255 (because I give an offset as u8 instead of sending the whole tick)

spring raptor
dire aurora
#

Problem is that if it isn't present you can't pass it to as a pointer, or you'd need to do MaybeUninit, but then you'd need to duplicate a branch that already exists in write into deserialize

spring raptor
#

I.e. you always pass MaybeUninit, then deserialize initializes fields.

dire aurora
#

You can, but you'd have to check if it is actually already initialized, and then execute alternative deserialize_in_place logic

#

But the write function already knows ahead of time if it's gonna be initialized or not, because it already branches on if it needs to create a new one or update it

spring raptor
#

Ah, right, something like Vec will require knowing if it's initialized or not

#

Will split it then. And maybe try transmute magic.
Going to sleep right now. But if you want to experiment with it - you are welcome to open a PR :)

dire aurora
#

Doubt I'll have time soon, I'm still working on refactoring my game to even work properly with the new rollback, and once that's done I need to start adding new rollback-related logic to make it work like normal again ferris_sob

#

Work like normal while still ignoring networking that is 😂

raw idol
#

this seems like the docs are wrong

image.png
#

also, will there always be two different client and server channel sets?

#

this is kinda bad for my integration

#

since I rely on having a single set of lanes on both sides

spring raptor
raw idol
#

actually I don't really understand how there can be two separate channel configs between the client and server

#

what does that even mean? how can you send along a reliable channel from the client to the server, and the server knows that it's a reliable channel that it receives on?

spring raptor
#

If it's a server event, it only sent from server to client.

raw idol
#

ok so in that case, if only the server sends you data,

  • client_channels: []
  • server_channels: [RepliconChannel { kind: Reliable }]
#
  • client has no valid channel indices to send out on
  • server sends out on channel index 0
  • client receives on index 0 -> maps it to server_channels[0]
#

hm

#

I think I can sort of hack this in

#

but oh man is this gonna be hacky

spring raptor
#

For client server channels are receive channels. For server - its channels are sent channels.

raw idol
#

yeah that makes sense I suppose

#

what I can do is merge the two channel sets - lanes = map_to_lane(client_channels) + map_to_lane(server_channels)

#

then map between lanes and client/server channels

#

or I can add support for send/recv exclusive lanes

#

I think I have some rewriting to do

spring raptor
#

I would suggest to try simplest integration possible even if it's a hack to get something working. You can always come back to reconsider :)

raw idol
#

well there's another problem

#

currently the message type is

#[derive(Debug, Clone, Message)]
pub struct RepliconMessage {
    pub channel_id: u8,
    pub payload: Bytes,
}
#

which provides an easy channel id to lane index mapping

#

but if I make channels and lanes use separate indices, then this breaks

#

so I have to add a sort of "lane factory" to the transport, which can decode lanes from messages using object-specific context

#

this lane factory is independent of the transport, but is dependent on the protocol (i.e. it's dependent on the fact that we're using a RepliconMessage)

spring raptor
#

Maybe you can create extra lanes?

#

Like for each server lane you create a bi-deractional lane, just use one direction.
Same for client.

raw idol
raw idol
#

if I'm a client and I receive RepliconMessage { channel_id: 4 }

#

RepliconMessage must have a function fn on_lane(&self) -> usize, which would give me 4

#

but 4 isn't actually the lane index, it's the channel index, which are now different things

spring raptor
#

But why they are different?

raw idol
#

and I don't have enough context in on_lane to remap that 4 to the real lane index

spring raptor
#

Ah, I get it

raw idol
spring raptor
#

Right...

raw idol
#

i.e. if you had client_channels: [a], server_channels: [b, c, d] then lanes: [a, b, c, d]

spring raptor
#

You can create a resource that maps lane id into specific channel id.

#

Two separate counters for client and server IDs. And you map them to lane IDs into a hashmap.

raw idol
#

my solution to this is something like

pub trait LaneMapperProtocol: TransportProtocol {
    type LaneMapper: LaneMapper;
}

pub trait LaneMapper<M> {
    fn read_lane(&self, msg: M) -> LaneIndex;
}

pub struct WebTransportClient<P>
where
    P: TransportProtocol + LaneMapperProtocol,
{
    lane_mapper: P::LaneMapper
}
raw idol
raw idol
#

this LaneMapper value will hold all the context required to map between lanes and channels

spring raptor
raw idol
#

not necessarily

#

it's a way to read the lane index of an arbitrary message

#

and in other scenarios, you might need extra context to read the lane of a message, which isn't contained in the message struct itself (like here!)

spring raptor
#

I thought that we talking just about integration crate

raw idol
#

oh, lanes are actually a transport-specific feature

#

so a transport isn't guaranteed to have lanes

#

ie the ChannelClient/Server don't use lanes, so we have no need for this remapping

#

it makes the internals more complicated (like here), but I think it's worth it in the end

#

because it's practically zero overhead

raw idol
spring raptor
#

I'm not familiar with your architecture, so I can't say much about it.
But in replicon we require channels, it's mandatory to request specific reliability guarantee from the underlying library.

raw idol
#

yeah I totally get that, and that's why if a transport doesn't support lanes, it's expected to support always being reliable-ordered

#

like MPSC is always reliable ordered, so it just doesn't need lanes

#

which means that the channels that replicon uses actually don't matter for the mpsc transport

#

basically I think I have an idea, but it'll be complicated

spring raptor
raw idol
#

how would that work with mpsc?

#

you can't get different guarantees with mpsc channels - they all have the same reliable+ordered guarantee

spring raptor
#

Yes, but channels should be present anyway.

raw idol
#

I disagree, there's no point for them to be there

spring raptor
#

I.e. you should be able to write struct A into channel 0 and struct B into channel 1.

#

And when you read channel 0, you read struct A.

#

Not struct B.

raw idol
#

channels with different message types is a completely different problem

spring raptor
#

It makes a lot of sense.

raw idol
#

I leave this up to the user

#

all messages sent through a transport use the same c2s/s2c types, no matter what lane

spring raptor
#

Usually it's not how it's done, libraries provide this abstraction on top of anything.

raw idol
#

I want aeronet to be fairly low level, just a thin wrapper around other apis

#

I leave the different message types to higher level libraries

spring raptor
#

Then you will probably will need an abstraction on top of it if you want to all your transports to work with replicon.
Channels will be necessary for this.

spring raptor
raw idol
#

that abstraction already exists, RepliconMessage is a byte buffer which is passed directly into/from RepliconClient/Server

#

because the client/server just deal with raw bytes, I only have to deal with raw bytes

spring raptor
#

But if you don't have channels in MPSC, for example, you won't be able to read specific data

raw idol
#

here's a snipper of the integration code

image.png
#

ClientEvent::Recv is a signal that the transport received a RepliconMessage

#

the message holds a payload: Bytes, which I pass straight to replicon.insert_received

#

sicne insert_received just takes a Into<Bytes>

spring raptor
raw idol
#

yep, also part of the RepliconMessage

# 
#[derive(Debug, Clone, Message)]
pub struct RepliconMessage {
    pub channel_id: u8,
    pub payload: Bytes,
}
spring raptor
#

Ah, for MPSC you have some additional layer?

raw idol
#

wdym by additional layer?

spring raptor
#

For renet we have a single integration crate, each transport works via RenetSever.
You probably write separate integration for each transport?

raw idol
#

the whole stack looks like

  • aeronet_channel: a set of (mpsc::Sender<T>, mpsc::Receiver<T>)
  • aeronet_replicon: makes a RepliconMessage, which you can use as T
  • your app: makes an aeronet_channel with T = RepliconMessage
#

let me send the replicon example code

# 
#[derive(Debug, Clone, Copy, TransportProtocol)]
#[c2s(RepliconMessage)]
#[s2c(RepliconMessage)]
struct AppProtocol;

/* derived by TransportProtocol:
impl TransportProtocol {
    type C2S = RepliconMessage;
    type S2C = RepliconMessage;
}
*/

type Client = WebTransportClient<AppProtocol>;
#

aeronet_replicon is abstract over all aeronet transports, that's the cool part about it

spring raptor
#

It's good, but you said that not all your transports have channels. So I assumed that for mpsc you will need some specific code.

raw idol
#

oh I see what you mean

spring raptor
#

Or you just include channel ID into each message even if transport have channels?

raw idol
#

no, the mpsc transport itself doesn't have the concept of channels - we move the concept of channels into the message itself

raw idol
spring raptor
#

Yes, I understand this part.

spring raptor
raw idol
#

the transport doesn't send the lane index separately, like the packet format doesn't look like [ack headers][fragment stuff][lane index][message payload]
instead, it looks like:
[ack headers][fragment stuff][message payload]
the [message payload] contains (an encoded form of) the [lane index], and the way it's decoded is left up to a trait OnLane.

For example, for RepliconMessage, the packet layout does look like [channel_id][payload]. But for something else, the type itself might be able to encode the lane index in a different more efficient way

spring raptor
#

Got it!

raw idol
#

one other thing to note, about the LaneMapper. What I mean by, "providing context", is turning this:

pub trait OnLane {
    fn lane_index(&self) -> LaneIndex;
}
// ...
let lane = my_message.lane_index();

into:

pub trait LaneMapper<M> {
    fn lane_index(&mut self, msg: &M) -> LaneIndex;
}
// ...
let lane = self.lane_mapper.lane_index(&my_message);

where the LaneMapper can hold any context that we need, e.g. for the replicon integration, which channel ID maps to which lane.

#

this can still be made ergonomic though, by effectively providing a "default mapper" for types which already impl OnLane:

impl<M: OnLane> LaneMapper<M> for () {
  fn lane_index(&mut self, msg: &M) -> LaneIndex {
    msg.lane_index()
  }
}
spring raptor
#

Makes sense

spring raptor
spring raptor
#

@dire aurora there is a small problem with inplace deserialization. In order to perform mapping, you need a read access to world, but when a component is mutably borrowed, you can't do it.

spring raptor
#

Also, splitting writing hurts ergonomics... Let's explore other approaches, maybe we can find something better.

For example, what about to leave it up to the prediction crate itself? I.e. you will provide your own serialize and deserialize functions and helpers to register a component with them for prediction/interpolation. And for users who want to also customize de/serialization at the same time, provide some generic helpers they need to call to have it working with prediction/interpolation.
Current serialize and deserialize are very flexible. Using them you are able to implement in-place deserialization if the component exists too.

raw idol
#

@spring raptor wondering if you know what could cause this? the input is really choppy on the client, but smooth on the server

#

it might be hard to tell in the video, but the box on the client looks like it's moving at 60hz, and on the server at 144hz

#

and yes, this is webtransport almost working with replicon :) (i still have to fix some stuff)

#

my reliable ordered protocol just doesn't work, and I can't connect to the server from a chromium browser, but other than that it nearly works

spring raptor
raw idol
spring raptor
raw idol
#

then I don't understand how it looks so smooth on the server

spring raptor
raw idol
#

ohhh

#

and the client doesn't simulate it?

spring raptor
#

But it send updates to clients on timer or fixed update, depending on how you configure it

raw idol
#

ok, so the client has to wait a full RTT before it gets its own position changed

spring raptor
#

Client don't do any movement, it only receives updates if there is no prediction.

spring raptor
raw idol
#

makes sense

#

but so far it's been really easy to integrate my transports with replicon apart from the lanes stuff

spring raptor
#

This is where third-party crates are needed. For a fast-paced game you will need something like bevy_replicon_snap or the upcoming prediction crate from @dire aurora.

raw idol
#

only thing I'd say is in Client ID::new, it should really be noted that you can't use 0 as the ID, as it's reserved for server

spring raptor
#

Makes sense. Feel free to open a PR to improve the docs if you spot something like this.

dire aurora
#

The only way you could combine it at that point is if both crates gave you custom methods, but then you'd still have the problem that things are per component now. So if I have interpolation and prediction and regular writes, the deserialize function would become some monstrosity that needs to check half of an archetype every component 😅

spring raptor
# dire aurora The only way you could combine it at that point is if both crates gave you custo...

Maybe you could do specialization to avoid it? I.e. register C, in group with Remote<C> and it will be a predicted deserialization functon.

With that approach you can't combine a crate that does serialization in a different way with a crate that writes things in a different way
Do you think that we will have crates with custom serialization? It's usually something that user provides if needed.
But even if will, maybe such crate could provide a generic hooks to which user could pass prediction logic. But it needs to be explored...

dire aurora
#

A crate like bundlication would be pretty much entirely custom serialization no? They could provide a way to inject a write function, but other crates wouldn't provide a convinient write function in the right format for any arbitrary crate that does something with serialization 🤔

spring raptor
#

What if replicon provide such API? 🤔

#

I.e. hooks

#

This way they will be in unified format

#

So split writing, but combine it with deserialization function on registration step.

dire aurora
#

Not to mention the extra overhead you'd get if every component handler starts checking the exact same marker components as all the other component handlers 🤔

spring raptor
dire aurora
#

Deserialization ID is irrelevant, since the server doesn't (and shouldn't) know about the existance of rollback histories and interpolation

spring raptor
#

With original writing separation that I pushed in the PR, writing function is also per component.

#

You need it per-entity for prediction, I understand that, but with specialzation you can achieve something like this by using a marker.

#

BTW, do you need per-entity for interpolation too?

dire aurora
#

I'd sure hope interpolation doesn't try to interpolate things without a marker component at least 😅

spring raptor
#

Sure 😅 I meant that do you need it for all components on the entity?

#

for all replicated components*

dire aurora
#

I think all the components it's registered for at least

spring raptor
#

So you don't like the suggested approach with splitting serialization and writing and just concatenate them at compile time when you register a function?
You want a true per-entity writing customization, right?

dire aurora
#

If we want good performance we'd need per-entity customization yea ... Or more specifically we'd need to filter write functions per-entity based on markers, then pass in the highest priority one that's available ... Like say someone spawns an entity with Replicated, Predicted and Interpolated, they probably expect the interpolation logic to only run only on unpredicted components (But ideally those plugins let the user specify the priority, in case it's the other way around)

spring raptor
#
  • if someone decide that some components shouldn't be interpolated, it will also work.
dire aurora
#

We only cache archetypes on the server, Predicted and Interpolated don't exist there, so they can't impact choosing the function

spring raptor
#

Like imagine you have a Money component on an entity, it's highly unlickely that it needs to be interpolated.

dire aurora
#

We could have archetype logic on the client, but that would get kind of confusing if write functions spawn new components

dire aurora
spring raptor
spring raptor
dire aurora
#

We could base it on the archetype id before any modifications got done, but I'm not sure if observers could later cause problems anyway

dire aurora
#

In the majority of cases components on Predicted would have a prediction write function, but there may be exceptions, like Money

spring raptor
dire aurora
#

Write function is technically per-component, but we can filter some of the registered write functions entirely on an entity level

#

Lets say we had 4 write functions and a default one, we could check the 4 markers, keep that as a list of booleans, then for any component we pick the first write function that has true

spring raptor
#

Ah, I think I get what you suggesting!
But we actually can achieve the same with specialization. You can provide a special marker on server and based on this marker register a group with a component and this marker.

#

I.e. even if you don't have Remove<T> on server, you can have a different marker that enables prediction.

dire aurora
#

We can't because what is and isn't predicted would vary per client

#

Which means we would need an archetype cache per client which would probably be bad in all sorts of ways 😅

spring raptor
#

Right...

#

It's more of a client thing, it needs a separate API

dire aurora
#

We could also try to keep an archetype cache on the client, but I'm not sure yet how future features could mess with that 🤔

spring raptor
#

So we need a way to intercept writing on client based on some sort of API.

#

How are you going to tell client that specific entity needs Remote<T>?

dire aurora
#

On the client side you'd know based on markers, otherwise the API for prediction/interpolation/whatever would feel very different from replicon

spring raptor
#

Yes, it makes sense, but how you send this markers?

dire aurora
#

We don't send them, the client would have it's own logic to figure it out ... With predicted in most cases the entity would also already be there before the server sends it, but for interpolation some system on the client would need to add it 🤔

spring raptor
dire aurora
#

If the server were to send a new entity, in most cases you wouldn't predict that entity, the user would probably write a system that adds Predicted when needed, kind of like those blueprint systems

spring raptor
dire aurora
#

In my case it would probably be something like With<CharacterController>, Without<Predicted> or With<MovingPlatform>, Without<Predicted> ... But with my old approach (which is much more common) I would only add predicted to the player, anything the client spawned as result of a prediction (like some vfxes), and maybe some things the player is currently interacting with (if you're modifying a voxel chunk, you'd predict that until the player stops modifying it and the server's state arrived)

spring raptor
dire aurora
#

Luckily clients do get less entities to deal with, so the archetype optimization isn't as important. Because ideally you only get updates for visible entities that changed

spring raptor
#

Okay, I will have to think about how we can achieve it a little more...

If you come up with something too - feel free to describe the approach, I will be happy to try experiment with it.

dire aurora
#

At the entity update level at least we could create a list of booleans for the presence of the marker component of each known write function. And for each component we could keep a list in the same order with Option<write_fn>, zip iterators over the both lists, filter out anything that's not enabled and grab the first ... But that would leave the challenge with writing ... Most significantly the entity mapping issue 🤔

#

Hmmm, looking at what client mapper needs world for I think we can do this in the same way I did it in bundlication ... I make a copy of Entities, spawn an id using it, add that to a list, and later finalize all entities in that list (in my case I add Identifier later, for replicon it would be adding Replication later)

#

That way using EntityWorldMut is save as long as no one calls Wolrd.spawn(), which I think write functions should not be doing in the first place

spring raptor
dire aurora
#

The list of what is and isn't allowed (based on markers) is per-entity, but the Option<write_fn> would be per component. So selecting the first option would also be per component

raw idol
#

it's still incredibly broken

#

but it kind of almost works

#

the WASM one looks completely broken

spring raptor
spring raptor
raw idol
#

there are moments where I'm not pressing any input keys and the wasm client moves on its own

#

also other clients don't seem to show up on the wasm client

#

in general it's just broken

#

also to connect to the server you have to manually type in its tls certificate hash into the "cert hash" box

#

because wasm + egui + clipboard doens't work

echo lion
#

Congrats :) you beat me to it, I've been working on that the past week

raw idol
#

btw, have you managed to ever get your browser to accept your server's self signed cert when working on your code?

#

this is literally my biggest struggle right now, no matter what I do I just get "opening handshake failed"

#

with zero error info of course

spring raptor
echo lion
spring raptor
dire aurora
#

You can just pass &Entities right?

#

It even has weird internal mutability iirc, so &mut is probably not even necessary

spring raptor
dire aurora
spring raptor
#

I.e. I borrow component mutably

dire aurora
#

You can use UnsafeWorldCell to create another pointer to World to borrow it from

#

Oh you can even take entities directly from UnsafeWorldCell

spring raptor
#

By Bevy internally

dire aurora
#

I don't think reserving entities should conflict with updating entities. But yea it's definitely not ideal. The nicer solution would be passing EntityMut instead of EntityWorldMut and some type of insert batch for anything new, that way you can't modify anything besides the entity's current components 🤔

#

But insert_by_ids is kind of horrible to work with, so that would be a bunch of extra work ferris_sob

spring raptor
dire aurora
#

Hmmm, yea we could just use a command queue ... Would be a bit slower, but inserts are slow anyway, and if it's just updating you can just use the EntityMut to update a component 🤔

spring raptor
#

And to reuse memory store it in Local<CommandQueue>.

#

Because we need manual flush

#

Commands are cheap to create, it's just a struct with refs

dire aurora
#

We would need to flush per entity or per packet tho, otherwise we run into updates if we find that entity again in another packet

spring raptor
#

Yep

spring raptor
dire aurora
#

We only order the functions, to be the same for every component, so it's cheaper to pick the right one ... Like a sparse array of function pointers

spring raptor
#

Sure, I mean that we sort it each time?

dire aurora
#

We just need to sort them at startup, just like how we have the groups ordered for priority

#

And then make sure we don't mess up that order

spring raptor
#

Maybe we talking about different things... I thought that you suggesting to zip entity components with list of functions. But we aren't always receive them all each time, right?

#

Like at first you may receive (A, B), but second update could get you only A.

dire aurora
#

No just the write functions with the bools, that way we can keep all the logic for picking the first one easy. It's just one iterator, it always returns something because there's a guaranteed to exist function at the end (the default write function)

spring raptor
dire aurora
#

We would have a list with the marker's component_id for each write function.. The list of bools could be a reused vec ... Something like this:

write_fns_enabled.clear();
write_fns_enabled.extend(markers.iter().map(|id| entity.contains_id(id)));
write_fns_enabled.push(true); // For default write
#

And the Vec<Option<write_fn>> could be stored in a place we already fetch I'd imagine 🤔

#

Since there's probably only gonna be 5 write functions even in the craziest projects that should be pretty cheap compared to hash map lookups

spring raptor
#

It's 2:17 AM for me, maybe I just not thinking well

#

For example, you not always interpolate all components, right?

dire aurora
#

Yea, a function would have to be called to register the write function for a component

#

If no interpolation write function exists for a component we can't use it, so we'd use the next option

spring raptor
#

Ah, I see

#

But I afraid that it still be a lookup into a HashMap

#

To check if such write function exists for specific component.

#

Or not? Writing function will be non-generic?

dire aurora
#

We would need to convert the write function to some generic type anyway, which is why Vec<Option<fn (idk) -> (something)>> would work

#

If write is the top-level function, and it calls deserialize, then I guess the function would already have a common function signature anyway, even if we were to use that transmute stuff for deserialize

spring raptor
dire aurora
#

Yea, which is why we need to register them per-component, even if they don't need special traits for T. We can't make Remote<T> unless we have T

spring raptor
#

Ah, I get it now

#

Not ideal, but should work...

dire aurora
#

Yea for Remote<T> it's kind of annoying, it makes a lot of sense for interpolation, which would need some trait to interpolate between the values tho

spring raptor
#

I also considered an approach with post-write callbacks. But also not quite good, I like it even less.

dire aurora
#

Oh like writing the value and having the crate in question do whatever it wants to return it? Yea that would be weird ... Could cause weird change detection problems

spring raptor
dire aurora
#

Yea there's a fairly decent chance it would get modified anyway when something is received, except when you check your predicted state against the new value and skip rollback if it isn't necessary

spring raptor
#

Ah, right

dire aurora
#

But the rollback crate could in theory try to handle change detection in a nice way too 🤔

spring raptor
#

I remember that Joy adviced to always rollback

dire aurora
#

Depends a bit on how and what you predict, tho sadly in the current state of bevy it's kind of necessary to do so, unless you want to heavily optimize your app

spring raptor
#

Then maybe post-write callback could simplify entity mapping and deserialization definition...
And in order to choose callback I can use the suggested system above, but for callbacks 🤔

#

But it's a bit weird

dire aurora
#

Yea it sounds kind of weird, things would also need to store the values constantly because replicon would overwrite them, probably would end up creating a ton more overhead than calling a write function and throwing anything that isn't an update in Commands

spring raptor
#

Another option to consider is to "let replicon know more".
I.e. instead of generic writes, provide built-in support for component history and cached value (for interpolation). Not quite sure how it will look or actually help us, just thinking out loud.

Your suggestion about array of bools + commands is workable solution, just thinking if we can make it a little more ergonomic.
Sometimes providing a less abstract API could improve it. Like we initially planned an API for archetypes, but ended up with groups to make API nicer.

spring raptor
dire aurora
#

The component history toggle could probably be as simple as checking if the selected write function wants old values, and if it doesn't but the value is old we only deserialize the value (so the cursor moves to the correct position) and discard the result

spring raptor
#

Yes, probably a good idea.

dire aurora
#

We'd also need some kind of general ID for all write functions that are for the same thing ... I guess we could have a function that registers the priority, marker component, and if it wants old values, and then have that return that ID ... Or if we want it to be less plugin order sensitive, we could take some write function group trait as an argument when registering write functions

spring raptor
#

Okay, I will experiment with it tomorrow, going to sleep right now

I would give this one a shot too: #1090432346907492443 message. Maybe it could be a little faster to fill this components for you on replicon side.

Discord
Discord - A New Way to Chat with Friends & Communities

Discord is the easiest way to communicate over voice, video, and text. Chat, hang out, and stay close with your friends and communities.

spring raptor
#

@dire aurora Can the same entity have some components predicted and some interpolated?

dire aurora
#

In theory yes, that specific combination is a bit weird but having a Remote<A> and RollbackHistory<B> could make sense

#

In that case whichever has higher priority should be used whenever available (probably predicted), falling back to the next option when no function was registered

spring raptor
#

Maybe it could be a little faster to fill this components for you on replicon side.
Not a good idea, I think what you proposing is better.

#

On second thought...
When we know that we have only 3 cases (write into something like Replicated<C>, History<C> or just C), it could be more convenient.
And to detect it we can fetch a single enum-marker instead of 2+ markers. Also no arrays, only two additional optional fields when you register a component or a group.

But with total abstraction the API becomes less intuitive... First, you register a marker via something like this: App::add_write_marker and when you register components or group of components, you need to pass marker functions in some specific order. To avoid ordering issue we can implement a trait on a function to return marker and then just sort by priority, but it's so inconvenient and ugly.

#

@dire aurora ^

dire aurora
#

It's not quite that simple tho. The logic isn't just "I want one old value" or "I want a list of old values"

#

Every crate that needs networked values will have different rules, which means replicon would need to support all those rules

spring raptor
dire aurora
#

For rollback you need a fixed number of items, determined by the configured number of rollback frames (which could possibly change on the fly), but it also always needs to keep the newest value that's outside the history

spring raptor
dire aurora
#

For interpolation you might also need somewhat of a history, since you'd need two states to interpolate between, and both of them might have arrived in the same frame

dire aurora
# spring raptor If I just provide a history, can't you after replication do anything you want in...

Yea they can do anything they want with it but there's a few issues: We have to copy things over into crate-specific components unless we want every crate that does something with replicon to be replicon exclusive. There would need to be some rule for what is kept in history, and it could get complex or non-optimal because of things like rollback needing the newest value before the history ... And the priority problem still exists, but now needs to be handled by the end user and the lower priority crate in the form of some marker component to disable each individual component where the higher priority one does something

spring raptor
dire aurora
#

In a way it isn't very ergonomic, but in a way it's also the most ergonomic option, since it's conceptually easier to understand for end users, and easier to make crates for that might have less common functionality

spring raptor
dire aurora
spring raptor
dire aurora
#

Instantiating the write function is done by some call to register a write function for a component, just like how custom deserialize functions look

#

The end user would need to register their component with the crate that does something with it anyway, can't interpolate components you didn't register systems for after all

spring raptor
#

And it won't work with manual implementation for a group.

#

User will need to manually create a Vec with the correct order.

dire aurora
#

Users don't pass the Vec, every registration is 1 function at a time

#

So a call to register a predicted component calls the function to register a predicted write function for that component

#

Same for interpolation

spring raptor
#

This is why I assumed that we need to group them somehow to send only a single ID.

dire aurora
#

We don't need to send an additional ID, since deserialize functions are for a component, we could store components in a separate list and point to the index in that list for each deserialize function

#

The server doesn't need to know anything about this whole system after all, it can just live entirely in the client

spring raptor
#

Not a function index, but an index to component write functions.

dire aurora
#

Yea, an index to where the write function Vec can be found

spring raptor
#

Got it, I think it makes sense.
Sorry if I asking a lot of questions, just trying to brainstorm the design :)

#

The only issue is left then is writing splitting.

I will try to play with it, maybe I will figure something out. I'm fine with transmut internally, but for public API it's not quite good.

dire aurora
#

If we can have the transmute internally do a type check to make sure the TypeId of the type you're passing in matches the one it was registered with we could make the public API fairly safe (assuming the transmute even works ofc)

spring raptor
spring raptor
#

Working on it, need to restructure a few things internally to make the suggested API look nicer.

shrewd sundial
#

Potential RTM question but I'm updating replicon to the latest version and has_authority() seems to need a parameter where it didn't in an earlier version.

.add_systems(
        PreUpdate,
        (
            client_init_physics_object_system.after(ClientSet::Receive),
...
            server_init_physics_object_system
                .run_if(has_authority()) /* Requires parameter now */
                .after(ClientSet::Receive)
                .after(client_init_physics_object_system),
        ),

The ultimate goal this code is attempting to achieve is to add physics components only to the servers version of the entity

raw idol
#

you need to pass the function itself, not the result of the function call

#

iirc

shrewd sundial
# raw idol `.run_if(has_authority)`

Did the trick, thank you very much!

.add_systems(
        PreUpdate,
        (
            client_init_physics_object_system.after(ClientSet::Receive),
...
            server_init_physics_object_system
                .run_if(has_authority)
                .after(ClientSet::Receive)
                .after(client_init_physics_object_system),
        ),
spring raptor
raw idol
#

if replicon tries to send a message along a reliable channel, but the underlying transport can't send that message, how does replicon respond?

#

i.e. if the message is too large or something, and the send function returns an error

spring raptor
raw idol
#

right now I'm just strongly thinking about what guarantees actually exist with these APIs, and making sure to avoid infinitely growing message buffers and DoS via memory exhaustion

#

and reliable messages make it really hard

spring raptor
echo lion
#

It’s reliable within a reasonable upper limit on throughput, hard to ask for more than that.

spring raptor
#

@dire aurora yes, transmute works! Right now all tests pass :)
It improves the API a lot. Now if user just want to specialize serialization, no unsafe needed.
For writing functions it's unavoidable, but I expect them to be defined by crates.

But don't look into it yet. I still need to add markers customization and a function for in-place writing. Should be easy, everything else is prepared, including commands and functions separation. Planning to finish tomorrow.
I will ping you when I'm done and provide a high-level PR description. But I basically doing what we discussed.

dire aurora
#

It would of course still be unsafe, since passing in B when the function is A would be bad, but it would be great if you at least got errors if you did anything besides the passed in type wrong 🤔

spring raptor
raw idol
#

networking is really hard, hard enough that we define strong guarantees for ourselves to make our lives easier. If one of these "reliable guarantees" isn't really a guarantee at all, everything falls apart

#

I would much much much rather tear down the entire connection rather than let a reliable message not be delivered in a reasonable amount of time

#

just to be clear, this isn't replicon's fault at all, but I think we should try to make these guarantees much stronger

spring raptor
# raw idol I'd argue that a reliable channel is either reliable or not; if a reliable messa...

Perhaps we misunderstood each other. My message was about error handling in general. For example, in unreliable channel Renet simply discards huge messages with a warning.
I assume (but I haven't looked closely at the code) that for a reliable channel this case will simply cause disconnect.

BTW, in replicon we also manually split message into packages, so we are unaffected by the mentioned "huge messages" problem 🙂

#

@dire aurora Looks like there are some issues with transmute...
I just tried adding deserialize_in_place and it causes memory issues.
I intentionally made it without calling deserialize to debug the problem. I even rewrote EntityMut creation to safe API.

What "fixes" the problem is this: https://github.com/projectharmonia/bevy_replicon/pull/227/commits/61c3150c5234ba6ec705cf56342dc0208b9ced67
I.e. I just inlined the logic of the helper function. So I don't think it's an actual fix.
Going to re-read the related Rustonomicon chapter.

raw idol
spring raptor
spring raptor
dire aurora
#

Can we store TypeId there and have a debug assertion that it's the same to prevent that from happening during development? 🤔

#

And I guess we could also do a trait bound on there? C: Component shouldn't match Mut I think? nvm you already said that

#

And afaik we can't register non-component deserializers

sharp roost
spring raptor
spring raptor
#

@dire aurora updated the PR: https://github.com/projectharmonia/bevy_replicon/pull/227
I still need to write tests and polish docs (that's why it's still a draft), but the code and examples in docs work!
Feedback is welcome.
Going to sleep right now, will be able to continue it tomorrow :)

GitHub
Marker-based API by Shatur · Pull Request #227 · projectharmonia/be...

This PR have multiple purposes, but unfortunately they highly related, so I can't split it into multiple PRs.
Right for prediction crates like https://github.com/Bendzae/bevy_replicon_snap or h...

dire aurora
#

RepliconTick is supposed to work like FrameCount but for the ticks of the simulation we're networking right? And I assume it's sent with each update to tell the client what tick that data belongs to so the other side can decide if it's new or old?

spring raptor
dire aurora
#

Then could we pass that to deserialize so we can use the tick to compress some data? I have lots of places where I just send an X ticks in the past value instead of the actual tick it happened ... Also might make sense to put fields like this in some "Context" struct so we could add things later without breaking the function signature for every serialize/deserialize function 🤔

#

The other obvious candidate for "Context" would be the mapper I think

spring raptor
dire aurora
#

Well basically I have a component, say LastGround(Tick). For the purposes of this component it doesn't actually matter if the tick was 10 ticks ago or 82892, all we need is the range of a u8. So it can be serialized as (current_tick - value).max(255) as u8, then deserialized again by doing (received_tick - value as u32) (with deserialize_in_place you can even improve it by keeping the current value if the value is 255 and the current value was more than 255 ticks old)

spring raptor
# dire aurora Well basically I have a component, say `LastGround(Tick)`. For the purposes of t...

Got it, makes sense.

What do you think about renaming ClientMapper into ReplicationMapper and put RepliconTick inside? Just to reduce the amount of nesting. ClientMapper right now looks like this:

pub struct ClientMapper<'a, 'w, 's> {
    pub commands: &'a mut Commands<'w, 's>,
    pub entity_map: &'a mut ServerEntityMap,
}

If I just add ReplicationCtx and put mapper inside, in write you will need to construct mapper first and then pass it into ReplicationCtx with the tick:
https://github.com/projectharmonia/bevy_replicon/blob/66e54ad12f380f83a325491634311de6214394d4/src/core/replication_fns/command_fns.rs?plain=1#L202-L205

I also considered passing ReplicationCtx into write inself instead of constructing it inside, but using commands will be awkward.

What do you think?

GitHub
bevy_replicon/src/core/replication_fns/command_fns.rs at 66e54ad12f...

ECS-focused high-level networking crate for the Bevy game engine. - projectharmonia/bevy_replicon

dire aurora
#

Hmmm, that constructing ClientMapper thing is pretty ugly ... Maybe we can have different context structs for write/serialize/deserialize and have ServerEntityMap for write context, and swap it out with a complete context for deserialize functions?

#

We also probably wouldn't want commands to be pub for deserialize functions 🤔

spring raptor
# dire aurora Hmmm, that constructing ClientMapper thing is pretty ugly ... Maybe we can have ...

Hmmm, that constructing ClientMapper thing is pretty ugly
Yeah... It just need commands in order to spawn, but commands also needed for insert :(

Having 3 different context won't be convenient... Also serialize will have only RepliconTick in the context.
Maybe it worth to just pass RepliconTick as usual.
And I will hide mapper fields, will be constructed with new when call deserialization.

dire aurora
#

Probably still useful to add a context to serialize, we might want add something like the last acknowledged tick later if we want to allow more optimizations

spring raptor
#

Nah, it's weird, let's use contexts and I will provide From to easily pass it from write to deserialize.

#

But I will do it in a follow-up PR, this one already quite big. I will do it right after merge.

#

(tick on serialize will require slight changing functions on server, I don't pass the tick there)

spring raptor
#

@dire aurora just finished with tests. Do you agree with everything else? If yes, I will mark it as ready for review.
After it I will submit the suggested follow-up about context and I think we can draft a new release.

#

I will implement With and Without filters later, going to open an issue about it.

dire aurora
#

Haven't looked super in detail at it yet but it seems correct ... Tho the comment on register_marker_fns seems a bit wrong, it uses the history itself as a marker, but the write function tries to spawn the history if it isn't present ... There's also some safety blocks that seem to have been copied and mention a write_history function that doesn't exist in that scope 🤔

#

Also don't see any code for the history vs latest value mode you mentioned before, tho I might've missed it or maybe it's planned for a follow up PR

#

Once this is merged I should also be able to relatively easily port bundlication's API and wire up the rollback stuff, so I could do some testing before the actual release gets pushed

spring raptor
spring raptor
spring raptor
echo lion
#

@spring raptor reviewed

dire aurora
#

Hmmm ... I think some of the fields on ReplicationRule should be pub so people can actually test their GroupReplication impls ... I think I'd need those getters for priority and components, and whatever is necessary to fetch a SerdeFns from ReplicationFns

#

Besides making new test and passing in a fake Tick value I think I have bundlication's API matching with GroupReplication now ... It's a pretty big commit tho 😅

26 files changed, 262 insertions(+), 4104 deletions(-)
spring raptor
spring raptor
dire aurora
dire aurora