bevy_replicon

[echo lion]

@spring raptor found a failure mode while trying to benchmark strings, made a pull request to your branch

[spring raptor]

It just fails to replicate?

Or panics?

[echo lion]

It also fails on master, idk what the deal is

Fails to replicate then the test panics at assert_eq!(app.world.entities().len(), ENTITIES);

[spring raptor]

Then it's "fine" 😅

Renet can't handle big messages.

It will print warning in the next release.

[echo lion]

It should work fine over localhost

[spring raptor]

https://github.com/lucaspoffo/renet/issues/123

It can be reproduced with Renet alone

Hm, the issue about unreliable

But it fails with reliable messages too

When you have to many parts.

[echo lion]

Hmm maybe a hard-coded limit in renet, ok

In any case, I reproduced your observation that the new PR is significantly slower even with large strings.

I am suspicious that Bytes::from_iter(buffer.iter_ranges(message_ranges)), is copying one byte at a time instead of a series of memcpy on each range.

[spring raptor]

Could be it...

I think I can try to profile it later (a little busy at the momoent)

But if it's the case, we better rollback to the previous solution.

[spring raptor]

On second thought, we already use Bytes::from_iter for updates just fine.

[echo lion]

Perhaps the Bytes internal buffer is reallocating capacity more often because writes are smaller

Which is even worse for larger entities because the total size is higher (additional doublings)

[echo lion]

Yeah looks like the culprit, testing now

[echo lion]

Hmm now with better allocation the perf is about equivalent between master and the single buffer

[spring raptor]

Looked into it, good catch.
But even if it's on par, probably worth use multiple buffers? It will simplify the code a lot.

[echo lion]

I am digging into the perf trace to see what's going on. Got cargo-instruments working on my mac.

[echo lion]

Looks like serialization is a very minor contributor to cost. Here is the server's replication sending system with the usize benchmark (update send benchmark only):

31.00 ms  100.0%    0 s     bevy_replicon::server::ServerPlugin::replication_sending_system::

27.00 ms   87.0%    0 s      bevy_replicon::server::collect_changes::
8.00 ms   25.8% 0 s       hashbrown::map::HashMap::insert::
4.00 ms   12.9% 0 s       bevy_replicon::server::replication_messages::InitMessage::end_entity_data::
4.00 ms   12.9% 0 s       bevy_replicon::server::replication_messages::UpdateMessage::end_entity_data::
2.00 ms    6.4% 0 s       hashbrown::map::HashMap::get::
2.00 ms    6.4% 0 s       bevy_replicon::server::replication_messages::UpdateMessage::start_entity_data::
1.00 ms    3.2% 0 s       bevy_ecs::component::ComponentTicks::is_changed::
1.00 ms    3.2% 0 s       bevy_ecs::component::ComponentTicks::is_added::
1.00 ms    3.2% 0 s       bevy_replicon::server::replication_messages::InitMessage::write_component::
1.00 ms    3.2% 0 s       bevy_replicon::server::replication_messages::UpdateMessage::write_component::
1.00 ms    3.2% 0 s       bevy_replicon::server::replication_messages::ReplicationMessages::iter_mut_with_info::
1.00 ms    3.2% 0 s       bevy_replicon::server::collect_changes::
1.00 ms    3.2% 0 s       core..iter..traits..iterator..Iterator$GT$::next::

4.00 ms   12.9% 0 s      bevy_replicon::server::replication_messages::ReplicationMessages::send::
2.00 ms    6.4% 0 s       bevy_replicon::server::replication_messages::InitMessage::send::
1.00 ms    3.2% 0 s        bevy_replicon::server::replication_messages::InitMessage::send::
1.00 ms    3.2% 1.00 ms         alloc::vec::Vec$LT$T$C$A$GT$::extend_from_slice::
1.00 ms    3.2% 0 s        alloc::vec::Vec$LT$T$C$A$GT$::extend_from_slice::
2.00 ms    6.4% 0 s       bevy_replicon::server::replication_messages::UpdateMessage::send::
1.00 ms    3.2% 1.00 ms        core::slice::::iter::
1.00 ms    3.2% 0 s        core::iter::traits::iterator::Iterator::sum::

The big expenses are inserting/getting stuff from ClientInfo hashmaps, and serializing entities.

[spring raptor]

Interesting!

It's quite surprising that hashmaps costs that much.

Maybe stop reusing the memory in places where we use HashMaps?

[echo lion]

How would that help?

[spring raptor]

If reusing memory costs less, it will be a perf win

[echo lion]

Maybe we can do an intermediate approach instead of one big buffer. Each 'write chunk' is initialized to a shared write buffer the first time it is needed by a client, then copied into individual client buffers. This way we only need to serialize entities once, which appears to be a decent size cost. But no need to cache and wrangle ranges.

[echo lion]

I think you contradicted yourself there, reuse or don't reuse memory?

[spring raptor]

Yes, sounds good to me!

[echo lion]

We never update buffers out of order, so the 'shared write chunk' will always be fresh until reset with a new value.

[spring raptor]

Right, I wanted to say if allocations costs less then lookups, then getting rid of them could save us some lookups. But then I remembed that our cache stored in regular Vecs, so never mind.

[echo lion]

I think the alternative is my VecDeque<Option<>> idea. But then we have to think about attack vectors again.

[spring raptor]

Let's do this: I refactor the current branch into using multiple buffers as before, but keep the architecture (logic inside messages). It's a good change anyway.

Then in separate PR I will try to copy and see how much performance it will get us.

[echo lion]

Btw I got a perf trace for structs and writing components is only ~ twice as expensive, but still < 15% of total cost. Glad I looked into this (never optimize blind!).

[spring raptor]

Agree!

Thanks a lot

[echo lion]

I redid the perf analysis, cargo-instruments was not playing well with criterion (it only runs one or two loops, so you don't get a high iter count, I had to set it manually). The HashMap::insert entry dropped out (no longer allocating nodes). Now HashMap::get() and UpdateMessage::end_entity_data() dominate.

595.00 ms  100.0%   0 s     bevy_replicon::server::ServerPlugin::replication_sending_system::

440.00 ms   73.9%   0 s      bevy_replicon::server::collect_changes::
132.00 ms   22.1%   0 s       bevy_replicon::server::replication_messages::UpdateMessage::end_entity_data::
107.00 ms   17.9%   0 s       hashbrown::map::HashMap::get::
44.00 ms    7.3%    0 s       bevy_replicon::server::replication_messages::UpdateMessage::write_component::
29.00 ms    4.8%    16.00 ms          bevy_replicon::server::collect_changes::
24.00 ms    4.0%    0 s       core..iter..traits..iterator..Iterator$GT$::next::
23.00 ms    3.8%    0 s       bevy_ecs::component::ComponentTicks::is_changed::
22.00 ms    3.6%    0 s       bevy_ecs::component::ComponentTicks::is_added::
12.00 ms    2.0%    0 s       bevy_replicon::server::replication_messages::ReplicationMessages::iter_mut_with_info::
11.00 ms    1.8%    0 s       bevy_replicon::server::get_component_unchecked::

7.00 ms    1.1% 0 s       bevy_replicon::server::replication_messages::InitMessage::end_entity_data::
6.00 ms    1.0% 0 s       bevy_replicon::server::replication_messages::InitMessage::start_entity_data::
6.00 ms    1.0% 0 s       bevy_replicon::server::replication_messages::UpdateMessage::start_entity_data::
6.00 ms    1.0% 6.00 ms       _$LT$core..result..Result$LT$T$C$E$GT$$u20$as$u20$core..ops..try_trait..Try$GT$::branch::
5.00 ms    0.8% 0 s       hashbrown::map::HashMap::insert::
2.00 ms    0.3% 0 s       core..Iterator$GT$::next::
1.00 ms    0.1% 1.00 ms       bevy_ecs::archetype::ArchetypeEntity::entity::
1.00 ms    0.1% 1.00 ms       bevy_ecs::system::system_param::SystemChangeTick::last_run::
1.00 ms    0.1% 0 s       bevy_replicon::server::replication_messages::InitMessage::write_component::
1.00 ms    0.1% 0 s       bevy_replicon::server::replication_messages::InitMessage::take_entity_data::

154.00 ms   25.8%   0 s      bevy_replicon::server::replication_messages::ReplicationMessages::send::

Ahah! But then if I do the test on master, write_component() dominates.

579.00 ms  100.0%   0 s     bevy_replicon::server::ServerPlugin::replication_sending_system::

478.00 ms   82.5%   0 s      bevy_replicon::server::collect_changes::
255.00 ms   44.0%   0 s       bevy_replicon::server::replication_buffer::ReplicationBuffer::write_component::
77.00 ms   13.2%    0 s       hashbrown::map::HashMap::get::
36.00 ms    6.2%    0 s       bevy_replicon::server::replication_buffer::ReplicationBuffer::end_entity_data::
25.00 ms    4.3%    0 s       core..iter..traits..iterator..Iterator$GT$::next::
22.00 ms    3.7%    0 s       bevy_ecs::component::ComponentTicks::is_added::
20.00 ms    3.4%    0 s       bevy_ecs::component::ComponentTicks::is_changed::
16.00 ms    2.7%    7.00 ms       bevy_replicon::server::collect_changes::
12.00 ms    2.0%    0 s       bevy_replicon::server::replication_messages::UpdateMessage::register_entity::
5.00 ms    0.8% 0 s       bevy_replicon::server::get_component_unchecked::
3.00 ms    0.5% 0 s       bevy_replicon::server::replication_messages::ReplicationMessages::iter_mut_with_info::
2.00 ms    0.3% 0 s       core..iter..traits..iterator..Iterator$GT$::next::
1.00 ms    0.1% 1.00 ms       
1.00 ms    0.1% 1.00 ms       core..iter..traits..collect..IntoIterator$GT$::into_iter::
1.00 ms    0.1% 1.00 ms       core..iter..traits..iterator..Iterator$GT$::next::
1.00 ms    0.1% 1.00 ms       bevy_ecs::system::system_param::SystemChangeTick::this_run::
1.00 ms    0.1% 1.00 ms       _$LT$core..result..Resultcore..ops..try_trait..Try$GT$::branch::

101.00 ms   17.4%   0 s      bevy_replicon::server::replication_messages::ReplicationMessages::send::

I am optimistic about an intermediate copy-buffer 🙂

[echo lion]

@spring raptor I think ReplicationMessages::send() has a bug with setting the last change tick. This code assumes if one init message is sendable, then they all are. However init messages will be sent if there is init data or if a client just connected. If just one client has an init message because it just connected then the last_change_tick invariants are broken. I'm not sure all the consequences of this but it needs to be thoroughly tested.

[spring raptor]

It's probably because write_component writes the entity 😅 On master it's done lazily.

[spring raptor]

Right!
I will look into it, thanks!

[waxen barn]

Does Renet automatically chunk data if it goes over the MTU, and if not is this something Replicon supports? Starting to look at Replicon again, I decided to go straight bevy_renet so I can just focus on implementing using a pattern i'm familiar with, but now that I have a better understanding of the problem I'm looking to clean up and Replicon seems appealing particularly around the server->client replication.

Also, at times I will likely need more fine grained control, is interfacing directly with Renet a bad idea when using Replicon?

Example: One thing I want to eventually do is before the server sends snapshots to each client, determine what clients should be seeing and only replicate that, basically networked fog of war. Is this type of control possible, in otherwords I need to send a different snapshot per-client. And if not, would something like that be possible eventually?

[spring raptor]

Does Renet automatically chunk data if it goes over the MTU
Yes.
But in Replicon for update channel (that is used for component changes) we do manual splitting to pack entities up to max packet size.

[waxen barn]

And does it replicate only changed values?

[spring raptor]

Sure, only changed components.

Also, at times I will likely need more fine grained control, is interfacing directly with Renet a bad idea when using Replicon?
It's possible, yes. But your channel ID should come after Replicon's channels. You will figure it out after looking at the setup.

[spring raptor]

This is a planned feature. Should be our next thing to add. See this issue about the implementation details.
https://github.com/lifescapegame/bevy_replicon/issues/15

[waxen barn]

How is that serialized? Like for example is there a 32bit Id per component or is it using some kind of bitmask. Basically my concern is egress as I'll be replicating a lot of entities.

Like would it be a bad idea to have a value per component so that I can only replicate values that change

[spring raptor]

When you register a component, it gets it's own ID (basically, an index). Then we serialize it using varints, so they are quite small, usually 1 byte.

[waxen barn]

Oh nice nice

thank you for answering these, I appreciate it

[spring raptor]

Sure, feel free to ask any questions about the impl.
Me and @koe put a lot of work to make it efficient :)

[waxen barn]

Hell yeah! Especially important if using cloud servers, egress costs suuck 😛

This Rooms feature sounds extremely awesome

[spring raptor]

We also have custom serialization for entities as two separate varints for generation and index to reduce the size.

[waxen barn]

Thats great, these features and the rooms feature would put you on-par with Unreal's networking features like the Replication Graph

thats amazing lol

One important thing to note is Fog Of War in many cases is more than visibility but also any actions being taken. For example, you have a minimap where enemies appear when they shoot, you also want to play the spatial sound from where they're shooting from, so you may want to at least replicate their transform at the moment of the action even if not visible, etc.

[spring raptor]

The planned rooms feature will work per-entity. Doing it per-component would be too costly.
So for this specific case I would include the position data into your action if it's done from an entity outside of visibility.

[waxen barn]

yeah that makes sense, and probably cleaner

I can't wait to try out replicon now that I have a better understanding of how renet itself works

And if you don't mind, do you two have games you're actively building this to use?

[spring raptor]

I working on a life simulation game:
https://github.com/lifescapegame/lifescape
But it's not ready, don't try it 😅
Going to focus on it more right after current buffer refactoring and the mentioned rooms.

[waxen barn]

amazing, thank you

[spring raptor]

The crate was originally part of my game, but @aceeri asked me to export it into a library long time ago. I'm so glad I did it, the code now in a much better state thanks to the Bevy community.

[waxen barn]

where the heck do you find the time 😛

[spring raptor]

I started long time ago 😅

[echo lion]

I think you’d need a custom solution for this, like replicating ‘sound zone’ entities which record sounds from a general area (rather than a specific location), or sending unordered reliable messages.

Actually you’d probably just spawn a sound effect entity on the server with customized location and visibility.

[echo lion]

Yeah I am also building a game. The WIP infrastructure and tech demo are at https://github.com/UkoeHB/bevy_girk and https://github.com/UkoeHB/bevy_girk_demo.

[spring raptor]

@echo lion okay, the rafactoring part is done, opened a fresh PR: https://github.com/lifescapegame/bevy_replicon/pull/158

[spring raptor]

I remember about it, will be as a part of a separate PR later.

[spring raptor]

It's ready for review, I will do the discussed copying in a separate PR.

[echo lion]

Got it, I plan to look later today

[echo lion]

@spring raptor done, sorry for the wait

[spring raptor]

It's okay, I'm not in a rush :)

[spring raptor]

I think that benchmark from https://github.com/lifescapegame/bevy_replicon/pull/157 is worth to have, right?
If yes, I will retarget this branch to master tomorrow (going to sleep right now).

[echo lion]

EDIT: I updated the PR on top of master

[candid eagle]

Quick question, What is the purpose of these functions (I can't infer from the return type :P) Is there an example of how I might use them?

[spring raptor]

It's built-in system conditions that you can use for run_if.
They are re-exported from Renet.

This is why they lack docs 😅

[candid eagle]

ah i see, that'd do it

[echo lion]

@spring raptor would you object if I add a PR to extend add_server_event_with() to include the queue system and local resending system? I'd like to reuse the queing logic but my setup doesn't use events.

Same with add_client_event_with()

[spring raptor]

Why do you need custom queue and local resending system?

[echo lion]

I don't use events, so I need the queue emptying to have special handling. I also want to completely disable the local resending system since I don't need it (this one is less important since it will do nothing if there are no events).

[spring raptor]

I don't get it. If you don't use events, why do you need add_server_event_with?

[echo lion]

I use it for setting up renet channels, it's slightly abusive of the replicon API lol:

• registering events: https://github.com/UkoeHB/bevy_girk/blob/d5918a403057249d5af38a7601f758ed13306b04/libs/bevy_girk_wiring/src/client_app_setup.rs#L131
• sending/receiving: https://github.com/UkoeHB/bevy_girk/blob/master/libs/bevy_girk_wiring/src/network_message_handling.rs

[spring raptor]

But why?
Just re-export create_client_channel.

[echo lion]

The problem is I need the replicon event synchronization, and the alternative is reimplementing it

[spring raptor]

And create_server_channel.

[echo lion]

Then I also need ServerEventChannel resource. It's basically reimplementing most of what's in replicon.

In any case, injecting queue_system is consistent with the API since receiving_system is injected.

[spring raptor]

It's already public

[echo lion]

I mean injecting the resource.

I'm trying to not reimplement everything that replicon is doing.

[spring raptor]

Custom receiving system make sense, but custom queue system is a bit odd...
Let's take a small step back and think about the goal. You need messages, but in form of messages, not in form of events, right? But you also want to hold messages until the specific tick like events?

[echo lion]

Yes

[spring raptor]

But why not events?

[echo lion]

1. I want FIFO because game/client messages can have ordering dependencies.
2. Bevy events are not good for networking because they can leak across reconnects.

[spring raptor]

1. We guarantee events order.
2. I think it can be fixed, right?

[echo lion]

1. Not order between different events.
2. Why bother, which I can just do what I'm doing now?

[spring raptor]

1. Do you have a predefined set of events, it's not something that user register, right?
2. Because we can do better.

[echo lion]

1. I use enums, and handle all messages strictly in order with an exclusive system.
2. Better than what, how? ?

[spring raptor]

1. I don't get it. If all your messages is a part of enum, why not register an enum event?
2. I mean instead of trying to hack the system, it's better to provide a nice API or try to fix downsides of the current system.

[echo lion]

I can't do one event because I need different channels for the event types (ordered/unordered/unreliable).

[spring raptor]

And you need all of them to be ordered?

[echo lion]

I want to handle all events in order, even if they come from an unordered channel yes

[spring raptor]

Hm... So if you receive something like 1,2,4, then you want to apply 1 and 2 and wait for 3 before applying 4 even for unordered events?

[echo lion]

No? I just apply whatever arrived each tick. (assuming it synchronizes with replicon, which I am trying to fix right now...)

[spring raptor]

Then I don't get about the ordering. Why do you need to order even from unordered channel and from an ordered one?

[echo lion]

Here's a problem with events: if you send a client event then it can get stuck in the event queue and be sent to the server on the other side of a reconnect (after reconnect logic has been applied). This is because events are not cleared until FixedUpdate has run at least once, and FixedUpdate may never run.

[echo lion]

Because that's what I decided for my architecture.

[spring raptor]

Why not just clear all events after disconnect?

[echo lion]

Now you are the one hacking the system lol

I'm going to eat dinner

[spring raptor]

That's an odd requirement. If I register an unordered event, I wouldn't try to order it with other events, but it's up to you of course...

[spring raptor]

Why? If you sent a networked event, then it is expected to drop it when you disconnect. It's just one additional line of code inside reset system, we can add it.

[candid eagle]

am i correct in thinking that you call app.add_client_event::<T>() on the client which sends the event and app.add_server_event::<T>() on the server? Or do i have it the wrong way around?

[spring raptor]

First one registers the event that will be transmitted from client to server. Second one will register an event from server to client.

[candid eagle]

yep, do you need to register on both ends or just one? If so which one?

[spring raptor]

You need to call each function on both ends.

[candid eagle]

ahhh, that makes sense

[spring raptor]

To clarify:

1. If you need an event from client to server, you call app.add_client_event::<T>() on server and on client.
2. If you need vice versa - app.add_server_event::<T>() also on both sides.

@echo lion If you don't want to use events - it's fine. We opened the door for user channels, we can adjust the API if you want, it would be reasonable.
You said that you want to reuse logic as much as possible, that's also undernstandable. But If you need the mentioned custom ordering, I don't think that you can reuse much from replicon's events.
So maybe just provide a better channels API? Or which parts you want to reuse for events?

[candid eagle]

yep, thanks! that makes a lot of sense, would you or @echo lion mind if i open a PR to add that to the docs?

[spring raptor]

Feel free to do so! I always welcome any contributions.

[echo lion]

Ordering everything together for consumption is just an artifact of putting all messages in one enum.

[candid eagle]

where is the source of truth for the docs? Do the docs.rs docs get maintained separately from the markdown in the github repo?

[spring raptor]

But you mentioned multiple channels. I thought that you mean that you want to ordering even between different channels, no?

[spring raptor]

In GitHub we have only readme. All docs inside the source code.

[echo lion]

There are channels for each of the event types (ordered/unordered/unreliable). I collect messages from each channel then handle them in one exclusive system (in order of when they were extracted from renet).

There is a trait for getting the send policy for each variant: https://github.com/UkoeHB/bevy_girk/blob/d5918a403057249d5af38a7601f758ed13306b04/libs/bevy_girk_game_fw/src/game_fw_msg.rs#L35

[spring raptor]

But you extract messages from channels separately.
So you will get ordered first, then unordered and then unreliable. Or in different order of channels, depends on which channel you read first.

[echo lion]

I will open a PR for this, it is probably necessary for the API to be correct

[spring raptor]

I thought that you trying to order events between channels and I curious why

Anyway, you mentioned that you don't want to use events and just use them to create a channel, right?
I think it will be just better to export methods from NetworkChannels for this case. If you need something really custom.

Going to sleep right now, it's 3 pm for me 😅

[candid eagle]

i've opened the pr for the docs change, only like two sentences plus formatting :)

[spring raptor]

I was going to sleep, but then I realized that you want an enum event, but on receive you want them to be separate events. Under ordering between channels you meant that you want all events from specific send-type to be ordered, not between reliable and unordered, for example, right? Then now I get why you want to reuse events. But I would suggest a simpler approach - just register your enum-event as usual, drain it on receive and send specific events in your system.

And we need to add a switch to disable local resending.

Now it's time to sleep :)

[echo lion]

I have a different idea. I will register renet channels separately for the three send policies, then add one replicon event channel with custom send/receive functions that push/pull messages from the three policy channels, and use a replicon event to consume renet messages. Hopefully this doesn't completely bust all my tests...

[candid eagle]

what is the earliest point after the client joins that they can send a client message?
I've got a postupdate system with .run_if(client_just_connected()) that sends an Ordered message to the server however the server doesn't recieve it,
the client is initialised in prestartup but that doesn't seem to be enough

[echo lion]

If it's in PostUpdate then it might be scheduled after ClientSet::Send. I would expect it to be sent the following tick, but maybe the way replicon drains the events queue messes it up. Try scheduling before ClientSet::Send

[candid eagle]

Ah ok, I'll try that when I'm at my PC next, thanks!

[candid eagle]

interestingly that didn't seem to fix it

[echo lion]

Are you sure the system is running? Did you get logs?

[candid eagle]

the event receiver system is running but the iterator doesn't have anything in it

hang on i'll add some tracing statements for timestamps

[echo lion]

On the server you will get a FromClient<T> event, not a T event

[candid eagle]

the event is constructed then stored in an entity (so i could send the event later to make sure renet was finished initialising)

that's the header of the reciever event on the server

that's the output from the instance running as a server client

which is the expected output

then if i join another client to it i get that on the client

and that' on the server (one extra line at the bottom)

so the connection gets established but the event doesn't make it to the server

[echo lion]

hmm

[candid eagle]

i'm positive that the event types are getting registered on both

[echo lion]

Do you maybe have a .run_if(has_authority()), bound somewhere?

I'm not sure what the problem could be. Hopefully @spring raptor can help when he wakes up (it is now my turn to turn in lol).

Everything you are doing looks right to me. One thing you can try is enabling logging for renet and replicon to dig deeper. They use the log crate, I'm not sure how to enable that with bevy's default logging. With tracing subscriber:

    let filter = tracing_subscriber::EnvFilter::builder()
        .with_default_directive(tracing_subscriber::filter::LevelFilter::TRACE.into())
        .from_env().unwrap();
    tracing_subscriber::FmtSubscriber::builder()
        .with_env_filter(filter)
        .with_writer(std::io::stdout)
        .init();

And use tracing-subscriber with features = [ "env-filter", "std", "tracing-log" ]

[candid eagle]

haha, thank you so much koe :)

[spring raptor]

Interesting idea!

@candid eagle could you share the code?
Or you could enable tracing for bevy_replicon like this:

.add_plugins(DefaultPlugins.set(LogPlugin {
    filter: "info,bevy_replicon=trace,
    ..Default::default(),
})

[candid eagle]

oh thanks, i was trying to get koe's snippet working but the log floods with gpu logs 😆

[candid eagle]

yeah i can share the code, its on gitlab but i can also just dm you a zip file

[spring raptor]

Try to make the code minimal, keep only necessary stuff.

[candid eagle]

there isn't much substantial stuff other than this netcode atm lol

[spring raptor]

I think you will need a custom queue system for it.
My suggestion is very similar, but instead of one event with custom systems and channels, you register 3 regular enum-events for each reliability type and then create a single system that drains the enum-events and does what you want.

If you agree with this suggestion, then you won't need extra ServerEventQueue functionality, so we can hide more:
https://github.com/lifescapegame/bevy_replicon/pull/164

[spring raptor]

Played with it, but due to the mentioned lazy writing I'm out of ideas how to implement it without turning the code into spaghetti 😅
So I'm going to put it aside for now. But if you have any ideas - feel free to suggest, I will be happy to try.

[echo lion]

You can add these to the env filter:

        .add_directive("bevy=warn".parse().unwrap())
        .add_directive("bevy_app=warn".parse().unwrap())
        .add_directive("bevy_core=warn".parse().unwrap())
        .add_directive("bevy_winit=warn".parse().unwrap())
        .add_directive("bevy_render=warn".parse().unwrap())
        .add_directive("blocking=warn".parse().unwrap())
        .add_directive("naga_oil=warn".parse().unwrap())

[echo lion]

I will try to find a solution

[echo lion]

I refactored to use my idea of separate renet channels + 2 replicon events. It looks like I won't need to mess with ServerEventQueue.

[spring raptor]

Curious, how it works?

Then yes, it's better hide extra things from public API. If someone requests it - we will reconsider to open it.

[echo lion]

• Register renet channels: https://github.com/UkoeHB/bevy_girk/blob/60304ab0b829311c56a14153df8d8b726e50f033/libs/bevy_girk_wiring/src/network_message_handling.rs#L42
• Forward game packets from ToClients<GamePacket> into renet channels (on server): https://github.com/UkoeHB/bevy_girk/blob/60304ab0b829311c56a14153df8d8b726e50f033/libs/bevy_girk_wiring/src/network_message_handling.rs#L67
• Take game packets from renet channels and send as <GamePacket> events (on client): https://github.com/UkoeHB/bevy_girk/blob/60304ab0b829311c56a14153df8d8b726e50f033/libs/bevy_girk_wiring/src/network_message_handling.rs#L96
• Register replicon events (on server): https://github.com/UkoeHB/bevy_girk/blob/60304ab0b829311c56a14153df8d8b726e50f033/libs/bevy_girk_wiring/src/game_app_setup.rs#L105

[spring raptor]

Interesting approach

[echo lion]

So replicon will intercept the client/server packets for server event queue extraction and client event buffer resetting, and my systems will do the rest (within replicon's scheduling).

[spring raptor]

But maybe we should implement explicit events ordering as an option...

Not sure how the API will look, will just keep it in mind.

[echo lion]

Thanks @spring raptor, glad we got all that sorted out.

[spring raptor]

I going to take a look at rooms tomorrow.

[echo lion]

Managed to knock a bunch of stuff off my todo list. But it's still the same length?!?

I will work on the shared copy buffer later today.

[spring raptor]

Cool!
Here is what I tried:

1. Returning slice of serialized data and store it into the Option outside. But works only for some things, like component, but doesn't work for entity data serialization since it's serialized dynamically. I can return an optional entity slice for write_component, but it's ugly.
2. Passing Option of slice into all functions. But it's ugly as hell. for write_component it's two Options - one for component and another for entity.
   Maybe you come up with something better 😅

[echo lion]

Ah, maybe figure out this bug before rooms? #1090432346907492443 message

[candid eagle]

thank you so much @spring raptor and @echo lion for helping me with the messaging problem,
as shatur deduced the problem was the client and server registering the events in a different order

[spring raptor]

My plan was to do it in rooms PR since that part has to be changed for them anyway, but I will split if it will be possible.

[echo lion]

Ok. It would be nice to have working tests for that bug before the rooms PR, which will be a big one.

If it makes sense to address it in the rooms PR, that works. I'd write the failing tests before-hand to document the current implementation (then comment them out or something).

[spring raptor]

Understandable, okay, I will try to fix it first and just include this commit into rooms PR temporarely (until your review, I think when it's day for me, it's night for you).

It's just 3 pm for me 😅

[echo lion]

Write the tests first, then decide if you want to fix it before rooms 🙂

You stay up late lmao

[spring raptor]

Makes sense! Will do. I will try to simplify the review process as much as possible.

[spring raptor]

Yes, my routine broke after the new year 😅
But I'm on vacation right now.

So I will able to work on it in the morning

[echo lion]

Boom: https://github.com/lifescapegame/bevy_replicon/pull/169

Running benches now

[spring raptor]

This is fast!
Will review tomorrow :)

[candid eagle]

am i right in thinking that you must implement MapNetworkEntities for all events that contain Entities regardless of whether they are client events or server events?

[echo lion]

Yes, if you want the server to consume client events with client entities that map to server entities. Replicon doesn't build that 'reverse' map for you though.

[candid eagle]

ah ok, so its only needed for client events not server events?

[echo lion]

It's needed for both, what I mean is the client can use ClientMapper to convert server events (e.g. how it is used in deserialize_mapped_component for replicated components that contain server entities), but you need to implement an equivalent tool for client events.

[candid eagle]

i see, is it always as simple as calling entity = mapper.map(entity) on each entity within the event?

[echo lion]

On the client yes. We get to use the existing [ server entity : client entity ] map created by replicon. Note that only replicated server entities will work with ClientMapper (well, obviously I guess lol).

[candid eagle]

yep, thanks!

[echo lion]

@spring raptor I think we are ~ at the limit of perf improvements in replicon without some advanced perf engineering. In my tests the replicon server takes about the same time as renet's encryption protocol, so I doubt we can get more than 5-10% total speedup, even with great effort.

@spring raptor The init benchmark includes all initial buffer allocations, should we change it so the test only looks at steady-state replication (or add a new test?)?

[spring raptor]

I think it would be good change to look at steady-state replication.

[spring raptor]

Yeah, we are pretty fast 😅

[spring raptor]

@echo lion opened https://github.com/lifescapegame/bevy_replicon/pull/170

[spring raptor]

But just to clarify - you don't do any of it. You only provide a trait and use a special registration, it will magically work.

You use it manually only when you need custom sending and receiving function.
For example, if you sending something that can't implement Serialize, like Box<dyn Refelct>. There are examples in the docs, but it's for advanced usage.

[spring raptor]

Looked into it. It updates last changed tick more often then it should. Doesn't cause issues, but should be fixed.
I will open a PR to track last change tick for each client separately + test. This will automatically fix the bug and needed for rooms anyway.

[spring raptor]

@echo lion Opened https://github.com/lifescapegame/bevy_replicon/pull/171, but without a test. But the current approach fixes it automatically, I don't think it's necessary.

[echo lion]

Best to add a test so we have more coverage of edge conditions.

[spring raptor]

The problem is this specific one kinda hard to add because the problem didn't have actual effect, it was just inefficient 😅
Going to sleep right now, if you have some ideas about it - feel free to send a PR.

[echo lion]

@spring raptor you can also detect if the change tick is the same as the previous message then clone the Bytes. Since clients are ordered by connection recency, only adjacent clients in the list will share a change tick so you don’t need to do any fancy search for a match.

[spring raptor]

I.e. iterate using windows()?
Will be away from the computer, would appreciate a PR :)

[echo lion]

Hey @woeful saffron I figured out your bug and should have a fix PRed later today 🙂

[echo lion]

@woeful saffron: https://github.com/lifescapegame/bevy_replicon/pull/173, can you try this branch?

[spring raptor]

@echo lion actually, I'm not sure if using serialized_size is faster...

It basically runs a serialization, just without allocations.

And counts size.

[echo lion]

hmm

[spring raptor]

If something is obviosly faster (like no allocations), it's good to optimize.
But if we aren't sure - let's pick the simplest way.

Because you know, without benchmarks it's hard to tell

[echo lion]

serialized_size should be faster than allocations for ints, but for arbitrary types it has arbitrary complexity

but at the same time, how many allocations are required to serialize an arbitrary type?

[spring raptor]

🤷‍♂️

It's not even a hot path

Damn, made a mistake in the PR, don't look 😅

Fixed

I pushed the combined solution. I think it should be okay now.
But let me know your thoughts about the allocation.

[spring raptor]

It doesn't pre-allocate upfront, but avoids creating Box and not runs serialization as serialized_size. Not perfect, but the simples from these 3 solutions.

[echo lion]

Oh nice 🙂 it fails on a race condition so maybe the timings changed in your case. I'm glad you didn't say anything or I wouldn't have looked at it again lol

[spring raptor]

Have you encountered this yourself?

[echo lion]

Yes, the test I added for 173 fails on master. (I did not experience it in the wild though)

[spring raptor]

But the test fails because we assume that tick should be inserted on any component insertion before.

It's doesn't actually triggers the behavior.

I mean it does, but because you explicitly removing the entity before.

[echo lion]

The test fails because we don't clean up client info updates after a despawn.

I had to finagle the test because otherwise we can't test the race condition.

[spring raptor]

Yes, I agree that the change is correct.

But let me try to trigger it, I really don't want to expose remove_despawned.

[echo lion]

Good luck 🙂

[spring raptor]

It's actually reproducible with a single line of code 😅

One extra update in despawn_update_replication test and it panics

[echo lion]

Hmm really? Update where?

[spring raptor]

In the end, one extra update for server is needed to trigger acknowledgment logic.

And it panics

[echo lion]

Ah the client sends an ack even if the update message is entirely ignored, that's what I failed to realize. I thought the despawn would cause the client to not send an ack.

[spring raptor]

Pushed the change to update the existing test instead of adding a new one and undid pub (I will do it in a separate PR)

Waiting for your response on https://github.com/lifescapegame/bevy_replicon/pull/173#discussion_r1451025034 and we are good to go.

I don't push on this, just want to clarify things for myself.

[echo lion]

Returns the tick recording the time this component was most recently changed.

We have a different use-case, so we should have a different name.

[spring raptor]

Saw it, agreed, approved 😅

[echo lion]

Can you make a release? Now that we are happily bug free 🙂

[echo lion]

The bevy compatibility table is out of data in the readme.

[spring raptor]

Thanks!

I need to automate it with cargo release somehow.

[echo lion]

Bevy is on 0.12.1 lol

[spring raptor]

Thanks, I probably should go to sleep 😅

The readme itself probably needs more improvements...
I think it lacks some details about what the crate supports. Yes, it mentions Renet, but it would be great to for newcomers to see that it supports auth, fragmentation, etc.

Maybe for the next release when we have interest management.
I will provide the mentioned draft tomorrow. It's almost done and since you will be online only at my evening, maybe I'll have time to write tests to showcase not only the interface, but prove that it works.

[echo lion]

@spring raptor another reason for room ids: you can send a server event to a room without allocating, but you can't send a server event to a group of clients without allocating.

[spring raptor]

Hm... What if we add concept of client groups? Basically rooms, but only for server events via SendMode::Group?

[echo lion]

I don’t think the server event and replication interest management should have different APIs. I imagine events and replication will almost always align based on logical rooms.

[spring raptor]

If you have rooms - yes.
But with the barebone interface - they are not related. For replication you configure which entities client see. For events - you configure which events.
The main benefit of the barebone interface is to don't pay for what you don't need. Room can be just added on top of it as part of separate plugin. Or game can implement specific API that more convenient for specific type of game.

[echo lion]

I mean practically speaking, many events will align with the logical rooms that entities are organized in.

[spring raptor]

Sure, I agree.
I just trying to say that for the barebone interface have distinct API could be fine. We just need to make sure that it's extensible enough to let users unite it if they want so.

I had a busier day than I expected... I didn't have time to finish the prototype. I'll try to finish it now and open a PR :) No tests yet, just to discuss.

[echo lion]

Ok 🙂

[spring raptor]

@echo lion opened: https://github.com/lifescapegame/bevy_replicon/pull/174
I provided description of my thoughts about it.
Feedback is welcome.

[echo lion]

For the boxed iterator you can make an enum of iterator types, then implement iterator on that enum. See the example at the bottom: https://depth-first.com/articles/2020/06/22/returning-rust-iterators/

[spring raptor]

Oh, cool!
Before opening the PR I also tried functional approach, like for_each, but it didn't play nicely with error-handling.

[spring raptor]

Thanks a lot for the feedback!
Then I will continue to iterate on the idea + apply the suggestions.

[spring raptor]

The PR is almost ready, working on tests.
I fill like we should split tests for replication granuraly into files because it becomes quite big. But I will do it as part of separate PR.

[spring raptor]

@echo lion the PR is ready:
https://github.com/lifescapegame/bevy_replicon/pull/174

[echo lion]

@spring raptor I have a high-level mode that tracks initialization state. I want it to have three subsections: Connecting (renet not connected), Syncing (connected and waiting for first replicon message), Init (connected and replication is synced). For robustness I want each mode to last at least one tick. To run the Syncing mode for at least one tick, I think I need to disable ClientSet::Receive when client_just_connected() is true (in my library, not in replicon). Do you think that would work/am I missing anything?

[spring raptor]

Yep, that what I would do.

Should work

[echo lion]

Actually, would need to disable in both Connecting and then Syncing for one tick in case of connecting in one tick (idk if this is actually possible, but theoretically...)

[spring raptor]

Yes, if you want at least one tick, it's better to do so.
Another idea would be to introduce states. This will make it automatically last for at least one tick due to how states work.

[echo lion]

That's what a mode is (bevy state)

[spring raptor]

Wait, do you mean Replicon tick, or Bevy's?

If Bevy tick, then it will work automatically for states.

[echo lion]

Bevy tick. The issue I am being careful about is 'skipping over' a state because multiple state transitions are satisfied at once.

[spring raptor]

Maybe you can put your state transitions into a single system?

[echo lion]

Although yeah, if I configure ClientSet::Receive.run_if(not(in_state(ClientFwState::Connecting))), then the first tick for ClientFwState::Syncing will not be preceded by a replicon refresh.

[spring raptor]

@echo lion working on readme update. Do you think it's better to use "intereset management" term or something more beginner-friendly?

[echo lion]

I quite dislike 'interest management'. Hmm..

How about 'visibility control'?

[spring raptor]

Looks like I need more tests to make sure that despawns are handled correctly.

[echo lion]

I am reviewing the PR atm

[spring raptor]

@echo lion maybe instead of bool store enum for clarity?

[echo lion]

Sounds good to me

[spring raptor]

Addressed all suggestions. Thanks!
Will write despawn tests tomorrow after work and I think it will be ready for the merge.

[spring raptor]

Not only despawn tests, there are many combinations that need to be tested 😅

[spring raptor]

While I was on my lunch break, I managed to address the suggestions and added some unit tests for ClientVisibility. But I will add more after work.

[spring raptor]

@echo lion done :)

[echo lion]

I will do another review in a couple hours. Also I figured out the Rooms design, it should be pretty easy (except for events, I'm not sure what to do here just yet although a RoomEventsWriter is probably required, then a system in PostUpdate that forwards events from the room writer to replicon).

[spring raptor]

Cool!
I suggested a possible solution for events in the rooms issue.

[echo lion]

Ok we are almost there 🙂

[spring raptor]

Addressed :)

[spring raptor]

@echo lion There is one inconsistency in events that bothers me.
We clear all events after client connection, but don't clear all events after server creation. Since server created right with the resource insertion we can't do the same thing.
I feel like this behavior could be unexpected for users.
Maybe just print a warning about event buffering if a client is not connected, but don't reset events before connection? This way user will still be notified about possible error, but the behavior will be consistent.

[echo lion]

Renet server will automatically drop events directed at disconnected clients. The reason we clear on the client is because the system that drains events into renet only runs when connected.

[spring raptor]

But if you create a broadcast event and later create a server - it will send an event to clients? While if you create an event while client is disconnected - it will be dropped.

[echo lion]

I guess we could try to handle this. I mostly assume the server is created on startup and then dropped at the end.

[spring raptor]

We can't drop server events before its creation because it will breaks singleplayer approach when you don't have a server :(

[echo lion]

You can write a custom run condition that checks for when the server was inserted. Then attach it to a system that clears pending server events. It's not zero-cost though, so it would be nice to hear a use-case.

[spring raptor]

It highly depends on the game. My game create server dynamically from a game menu, for example.

I'm not talking about use case of clearing events, just about consistency. It's a bit odd that we drop all client events before connection, but don't drop server events before its creation..

And I proposing to avoid dropping since we can't handle the server case nicely. Instead only warn users.

[echo lion]

Consistency isn’t really the issue for me, servers and clients have different designs and uses.

[echo lion]

No this is a bad reason to weaken client synchronization guarantees. Client reconnects are a normal part of clients. Recreating servers is not a normal part of servers (within the scope of a game).

[spring raptor]

Maybe clients and servers are, but events should work consistently.

[spring raptor]

How? If you send event before connection - you will be warned about it. It indicates a bug in the code. Just not discard it, that's the only change.

[echo lion]

If there is a bug and we know it’s a bug (because we are logging a warning), we should not just ignore the bug by failing to drop messages. Doing so willingly makes client code more likely to be broken in production, for only a stylistic reason.

[spring raptor]

Trying to recover after a bug is good, but we don't know if user wanted server to receive the event. Dropping it could also break something.
But without dropping it will be at least expected behavior. Bevy never drop events manually.

Just not discard it, that's the only change.
Not the only change. Need to run the system when client connecting too.

[echo lion]

The problem is you introduce ambiguity about the disconnect phase: the period of time when messages fail to send. That period of time always exists, because messages can fail to send within renet. The start of that period is unavoidably random and not immediately detectable (you need to wait for acks of some kind to know the status of a sent message). It is important to end that period cleanly so users can safely recover without headaches (eg messages hanging in limbo, without any guarantees or clarity about when or if they were sent).

A clean and precise networking API is way more valuable than one that over optimizes reconnects by introducing ambiguity.

[spring raptor]

Yes, we should reset after disconnect. That's expected. What I saying that we should let renet buffer messages instead of dropping them.

Sorry if I explained it poorly.

So what I suggesting is to move client event cleanup to disconnect and run event sending system even before connection and warn user if client is still connecting instead of dropping the event.

[echo lion]

It might be fine, I will think about it carefully.

[echo lion]

Isn’t it a problem that bevy events are only buffered for 1-2 ticks? Your idea can only work if events are drained into a temp buffer while connecting.

[spring raptor]

Ah, for some reason I thought that renet takes the message even if not connected. But it just ignores it in this case. Then never mind.

But I don't like the current cleanup before connection either...
Maybe we can reset after disconnect (so reconnects are still clean) and only warn if there are events before new connection without cleanup?

[echo lion]

Why would we not cleanup before connecting? Not cleaning up means submitted messages are in an ambiguous state even after connecting (you lose the clean sync point).

[spring raptor]

Because we don't know if user wanted the event to be received or not.
You don't lose the clean sync point because of cleanup after disconnect.

[echo lion]

If we can’t provide any delivery guarantee then it shouldn’t even be an option for users.

It’s like inviting people to write bugs.

[spring raptor]

That's how events work. If you forgot to receive it - you lose it after a few frames

[echo lion]

That’s over-mixing the concepts of bevy events and networking events.

With a bevy event you can write an event reader that is guaranteed to work. With a networking event that’s not the case if the event never even arrives.

[spring raptor]

Why so? The current solution is no different. I just works with a different assumption - that users didn't want server to receive an event.

[spring raptor]

The event reader does not help with missing events. It simply filters the events that you have seen on the current system.

[echo lion]

Unexpected behavior should not be exposed to users…

[echo lion]

?? A normal bevy event reader guarantees it will read all events if it runs every tick (assuming you don’t mess with the events resource).

[spring raptor]

I agree. But we can't do much about it.
In Bevy if you miss an event or send it in a wrong time - it's unavoidable user problem.

We can just make the behavior a bit more expected by avoiding manual cleanup. Just warn user about possible problem.

[echo lion]

In bevy you don’t miss events! What are you talking about

[echo lion]

Avoiding manual cleanup makes the behavior less expected. There are no expectations in that case, it’s a Schrödinger’s cat!

[spring raptor]

You won't miss an event only if your system runs more often then FixedUpdate. Otherwise you can easily miss it.
The purpose of event reader is to only read new events, it doesn't help with event misses.

[echo lion]

My point is if you want to guarantee you read all events, then you can do so easily. And if you fail to read an event, the reason is explicit and under your control.

Not some vague networking race condition footgun.

[spring raptor]

Avoiding manual cleanup makes the behavior less expected.
Manual cleanup after a disconnect is expected.

But why doing a cleanup in connect stage? If I send an event, I would expect a deliver attempt even if I not connected. So If there is a bug in user code, we need to warn about it, but proper "recover" for this bug is to keep the event in place.

You suggested that we should try to recover as much as possible after a bug to minimize its effect in production. And I agree with you.
But I'm not sure that proper recover is to drop the event in this case. I think we should warn, but keep it. With cleanup after disconnect, of course.

[echo lion]

Why are you so insistent on adding a vague race condition that in practice can only gain the user one or a few ticks where submitted messages might be sent? Why would we provide this ambiguity in the API as an option - “hey maybe you want to write code to target this race condition (it’s probably going to cause bugs though, good luck!)”.

[spring raptor]

Hm... I didn't think about it that way. You're right, leaving events in will make the behavior less predictable.
Okay, let's keep everything as is.

[echo lion]

Thank you 🙂

[echo lion]

Sorry if I have been annoyed and impatient. I’ve been working to get simple and robust networking APIs for months now and feel so close to being done…

[spring raptor]

It's okay :)
Your feedback is quite valuable, I like your fresh perspective on things.

@echo lion I planning to rename my game into "Project Harmonia". It's a placeholder name, I just not satisifed with the current one (Lifescape), but don't have a good replacement and want to emphasize that the name is temporary.
It will affect Replicon URL since it's in the game org. GitHub will create a redirect, but since I changing it, maybe it worth to move replicon into its own org? What do you think?

[echo lion]

Hmm not sure, I have not done much with GitHub orgs (need to figure that stuff out, I have a big pile of personal repos…).

[spring raptor]

Okay, we can keep everything as is for now.

[echo lion]

Should I PR rooms to replicon or do a separate crate bevy_replicon_rooms?

[spring raptor]

I think having bevy_replicon_rooms would be better. Keep only core stuff that can't be implemented outside in replicon and provide "addons".

[echo lion]

Here’s an idea: room combinators for events. You can say room_writer.send(Room1 + Room2, event) and the event will send to clients if they are in both rooms. A similar pattern might be possible for entities as well: replicate to clients in a group of rooms.

It could even be extended with OR and ANY combinators

[spring raptor]

Sounds nice!
But how it can be achieved? Some kind of macro to declare Room<number>?

[echo lion]

Not sure yet. I doubt it would need a macro though, just traits and some complex mapping in the rooms cache.

[spring raptor]

Would be interesting to see

[echo lion]

Hmm maybe ‘rooms’ is the wrong idea. A more general idea is ‘visibility attributes’. So an event or entity is replicated to a client if the client matches a ‘visibility attribute conditional’.

[waxen barn]

but wouldnt rooms also be good for things like chat and stuff

[echo lion]

Do you mean re:attributes? You’d give clients an ‘InGlobalChat’ attribute, etc.

[spring raptor]

Maybe... Something like Room(1) is less convenient then InGlobalChat.

[echo lion]

Attributes make a lot more sense for visibility conditionals: send event to clients who ‘(X or Y) and NOT(Z)’.

[waxen barn]

have you looked at rooms in Socket.io

[echo lion]

No, what are they doing?

[spring raptor]

Agree

[waxen barn]

https://socket.io/docs/v3/rooms/

its pretty simple

sounsd like what you guys are trying to do

simple for the user 😛

[echo lion]

Yeah something like that was my original plan, but then I got to thinking. Rooms on their own are not very flexible for complex visibility requirements. Just think of any MOBA and any invisible character. It makes more sense to use attributes than rooms to describe who can see that character.

The phrase ‘visibility attribute’ is kind of clunky, any better ideas?

[spring raptor]

I don't know, I like it

[echo lion]

Maybe bevy_replicon_visat? Less memorable than rooms 😦

[spring raptor]

I don't like it 😅

[waxen barn]

reminds me of vista 😛

[spring raptor]

@echo lion am I understand correctly, that instead of sending an event like this:

damage_events.send(DamageEvent { rooms: vec1[Room(1), Room(2)], .. } )

You want something like this:

damage_events.send(DamageEvent { visibility: AllVisibleHeroes, .. } )

[echo lion]

Right, more like that. Except visibility control as a method parameter not struct member.

I guess bevy_replicon_attributes would work

[waxen barn]

to be clear these are attributes relative to something, such as a player's camera

right?

[echo lion]

It can be anything. You assign attributes to clients, then write attribute conditionals for event/entity visibility. Attributes can be spatial or logical or whatever you want.

[waxen barn]

ahh nice yeah

[spring raptor]

Basically compile-time named rooms

[waxen barn]

bevy_replicon_context_attributes

long lol

[echo lion]

entity.insert((Fish, Replication, VisibleTo(Global + Not(IsClient(some client)) + Not(InATree))));

I’m not sure if a component-based or resource-based approach is better for registering entity visibility. With a resource you can easily iterate the entities that a client can see. With component-based you need to wait until the end of the tick to collect visibility changes.

On the other hand, an attribute-based approach implies decoupling of entities and clients so maybe iterating visible clients in the server isn’t that useful.

[spring raptor]

Agree

[spring raptor]

About the org. The benefit of dedicated GitHub org is a place to organize replicon-related crates. You can maintain your own plugins inside the org yourself, I maintain mine (if I will create any in the future) and we maintain replicon together as before.
Just an idea for the future, let me know when/if you want to organize your repos into an org, I will join.

[echo lion]

That’s a good idea, I will think about it

[spring raptor]

Started using dont_replicate approach and I think it's quite limiting. Sometimes I insert the component I want to ignore later.
I will try to rethink the API...

This was an easy change: https://github.com/projectharmonia/bevy_replicon/pull/177

[echo lion]

What was our reason for adding this limitation again?

[echo lion]

I think this reintroduces the bug: someone can replicate an entity with component A, remove component A, then later come back and add component A and ignore it.

[spring raptor]

To avoid removing the DontReplicate.

[spring raptor]

I will test it.

[spring raptor]

Right, it will be considered as added even if you remove it on the same tick:

        commands
            .entity(entity)
            .remove::<Transform>()
            .insert(Transform::default())
            .dont_replicate::<Transform>();

This won't panic, while it should.

This bug was present even before - remove Replication, insert it back and now you can call dont_replicate.
Need a better design, I will think about it.

[echo lion]

If you remove Replication then the entity will be despawned. Even bevy_replicon_repair handles this fine - if you miss the despawn, since the re-replicated entity won't have the ignored component, repair will remove the client's copy.

[spring raptor]

I think that the following will be counted as despawn and spawn at the same time and will result into despawn on client since spawns are handled first:

        commands
            .entity(entity)
            .remove::<Replication>()
            .insert((Replication, Transform))
            .dont_replicate::<Transform>();

This is so cursed 😅

[echo lion]

That sounds like a more general replicon bug, add a test?

[spring raptor]

Yep

[echo lion]

I thought RemovedComponents would only return entities that don't have the component. But maybe not

[spring raptor]

Let me quickly check it

[spring raptor]

Yep, the following assertion fails:

#[test]
fn respawn_replication() {
    let mut server_app = App::new();
    let mut client_app = App::new();
    for app in [&mut server_app, &mut client_app] {
        app.add_plugins((
            MinimalPlugins,
            ReplicationPlugins.set(ServerPlugin {
                tick_policy: TickPolicy::EveryFrame,
                ..Default::default()
            }),
        ))
        .replicate::<TableComponent>();
    }

    connect::single_client(&mut server_app, &mut client_app);

    server_app
        .world
        .spawn(Replication)
        .remove::<Replication>()
        .insert(Replication);

    server_app.update();
    client_app.update();

    assert_eq!(client_app.world.entities().len(), 1);
}

But we can fix it since we use our buffers 😅

This is very rare edge case, but I think we should fix it.
The same for components, need reinsertion test.

Going to sleep right now, will be able to address it tomorrow after work.

[echo lion]

Hmm concerning, I will have to look closer at my other uses of RemovedComponents

[spring raptor]

Maybe worth fixing on Bevy side...

[echo lion]

I spent all day working on visibility attributes. It started out fun then turned into a nightmare of rust lifetime issues. I'm pretty stuck right now: https://github.com/UkoeHB/bevy_replicon_attributes.

The end goal is to write conditions like this: VisibleTo::new(A && !B) with zero allocations for small numbers of condition nodes (that example is 4 nodes). To accomplish that my idea is to build a closure tree that can be traversed twice. Once to get the node tree size, then again to write the tree into a slice (either a heap or stack slice depending on size). However I cannot figure out how to manage the closure types and lifetimes to pass them into VisibleTo::new() cleanly.

There was also one lifetime issue when evaluating conditions that I gave up on and went unsafe.

I would love some help on this if possible.

Now that I think about it, it’s probably fine to do one pass over the tree. Just copy from the static buffer onto the heap if it’s too long. That should simplify things greatly, I’ll see how far it gets me tomorrow.

The worst case perf will be worse since there will be more allocations, since the full size isn’t known in advance (unless the compiler can inline everything and figure it out?).

[spring raptor]

There is also SmallVec crate that could help with it.

Another approach would be to create some sort of bitflags.

[spring raptor]

Looked into possible fix. I can switch from Vec<Entity> to HashSet<Entity> and remove all insertions from removal buffer, but it will have performance impact.

For components similar from Vec<ReplicationId> to HashSet<ReplicationId>. But I also need to iterate over the whole world because there is no other way to detect if a component by its ID was inserted. I can't do it on sending the message because I need to mutate this resource.

I'm not satisfied with the solution... What do you think?

[waxen barn]

hey have you folks gotten this to work with Bevy? https://github.com/lucaspoffo/renet/tree/master/renet_visualizer

at work right now so cant test it but just curious

[spring raptor]

Last time I tried (long time ago) it worked.

[waxen barn]

thats awesome

[spring raptor]

We also provide a plugin for diagnostic API in Bevy:
https://github.com/projectharmonia/bevy_replicon/blob/c890e378bf812d55a5cbec485bfc6ce4698b1e75/src/client/diagnostics.rs#L31

Example setup in the PR:
https://github.com/projectharmonia/bevy_replicon/pull/85

[waxen barn]

wow nice

[echo lion]

For replication we can do like dont_replicate and add a replicated() app extension. We’d need to add two components: an inner crate-only component that can’t be easily removed (you can remove by manually accessing the ecs), and an outer component for tagging the entity for user queries. Then there will only be conflicts if someone despawns and respawns an entity id in the same frame.

Alternatively we can order despawns and removals before spawns and insertions and just eat the duplication.

Technically it is more accurate to apply insertions/spawns last since they represent the most up to date entity state (we read them directly).

Btw thinking about value-diff compression. If we pass the change-limit tick + current tick into serializers, then the serializer can have a local that tracks historical state and performs a diff internally. The deserializer would also need to track historical state and apply diffs to the correct older value. Not 100% sure that works but seems promising.

[echo lion]

Might even be able to do this generically for types that implement certain traits?

It would require changes to the shared copy buffer, since different clients could get different component serializations.

[spring raptor]

This one would be more ergonomic and correct.

And for dont_replicate check just check if there is a removal?

[echo lion]

I have to get to my computer and check. I am constantly forgetting replicon implementation details lmao.

[spring raptor]

You are not alone in this 😅

I initially had issues with updating my game to dont_replicate, you suggested a possible edge case and it turned out to be a possible problem with all components 😅

[spring raptor]

Wait, no, if you spawn and despawn on the same frame, then client will have a leaked entity.
This is why spawns come first.

Actually, for components it evens out.
The issue only with spawn.
Writing a test right now to confirm.

[echo lion]

No, spawns will only be replicated if they exist in the world when collect_changes runs.

[spring raptor]

Right

Yep, probably a way to go.

So we need to restructure our init message like this:
0. Mappings

1. Despawns
2. Removals
3. Insertions

[echo lion]

We waste 6 bytes when there are only insertions sadly.

[spring raptor]

Yeah... But this will the the correct order.

[echo lion]

We could add one byte to the init message header for bit flags.

On average it should be an optimization. Most ticks won't have mappings so we already save one byte.

[spring raptor]

Agree

But I will submit a PR with the fix and tests fist.

[echo lion]

I'll be working on visibility attributes again today, hoping to make good progress.

[echo lion]

We still didn't solve this lol. I think the only solution is a replicated() app extension that inserts two components and locks the entity into being replicated.

[spring raptor]

Could you elaborate?

[echo lion]

This comment

[spring raptor]

Oh, got it.
I suggested a simpler solution.

Just check if there is a removal when you call dont_replicate

[echo lion]

We could do a replication builder:

commands.spawn((PlayerBundle))
    .replicated()
    .without::<Transform>()
    .without::<Hairstyle>()

[echo lion]

Does this work? What if the removal happened several ticks ago?

[spring raptor]

My initial problem was that I need to mark something as not-replicated later, when I insert T.

[spring raptor]

I think removals hold until the next fixed update, so it should be fine...

[echo lion]

Oh right I already commented this. Removing Replication isn't a problem

[spring raptor]

It's a problem, I provided a test that fails.

[spring raptor]

This one.

And it fixable by reordering data in init message.

[echo lion]

Ok so this is the issue: component removal within the current tick.

[spring raptor]

Yes, regular component and Replication both cause "leakage".

[echo lion]

I meant not a problem for ignored components.

[spring raptor]

Ah, yes.

I going to finish the reordering first and I will back to dont_replicate

I would consider it separate things.

[echo lion]

Ok I'm still waking up. The bug with your closed PR was that dont_replicate could be called on component removal -> insertion then replication will fail to remove the component from clients.

[echo lion]

And the bug still exists for the Replication component if you remove -> insert the component and call dont_replicate on insertion.

[spring raptor]

Correct.

Consider PR as a draft, I will back to it a little later.

[spring raptor]

@echo lion one quick tests reorganization PR: https://github.com/projectharmonia/bevy_replicon/pull/179

[spring raptor]

@echo lion opened: https://github.com/projectharmonia/bevy_replicon/pull/180

[echo lion]

done

[spring raptor]

Fun fact: 1/3 of the repo are tests 😅

[echo lion]

I got attributes to compile, not sure how to improve ergonomics yet: https://github.com/UkoeHB/bevy_replicon_attributes/blob/7f618e13c531b844d8634250da777abafd0a32c0/tests/test/tests.rs#L19

Maybe a macro?

commands.spawn((PlayerInventory, replicate_to!(IsClient(client_id) || IsSpecator)));

Like this? And then replicate_to!() will insert the Replication component as well.

[spring raptor]

Could work.

Have you tried overloading operators?

[echo lion]

To update the visibility, access the VisibleTo component and do:

visibility = visible_to!(Not(IsBlind) && Location(get_location(client_id)));

[spring raptor]

Hm...

A bit too magic as for my taste

[echo lion]

Not yet. The problem is the root attribute cannot be bare since I need to do this: https://github.com/UkoeHB/bevy_replicon_attributes/blob/7f618e13c531b844d8634250da777abafd0a32c0/src/temp.rs#L221

[echo lion]

My rust type voodoo is not powerful enough 😦

[echo lion]

Honestly without a macro we'd probably just do functions of the same names: replicate_to() and visible_to().

[echo lion]

My macro voodoo is even worse than my type voodoo so... we will see how far I get

I will move ahead with the existing API then go back to improve syntax if possible. Maybe recruit a macro wizard to help 🙂

[spring raptor]

Can't suggest much, not a wizzard myself 😅

[spring raptor]

How Location(get_location(client_id)) will work internally?

[echo lion]

Location would implement VisibilityAttribute, which has an inner_attribute_id method. You'd need to translate location-data into a u64.

[echo lion]

Oh @spring raptor I think I commented on the wrong one

[spring raptor]

The message looks related:
https://github.com/projectharmonia/bevy_replicon/pull/180#discussion_r1456669186

[echo lion]

Oh I misread the test and forgot we no longer allow removing Replication. Ok that comment can be ignored, the PR can be merged.

Wait no backtrack again, we can remove Replication but not reinsert it. So this test should be changed to remove Replication. That way if RemovedComponents behavior changes based on if the entity was despawned, this test won’t have changed behavior from what it’s testing.

[spring raptor]

Right, let me open a new PR for it

[echo lion]

Thanks, sorry about that

[spring raptor]

No need to sorry, It's a good suggestion. I the one who should be sorry for the oversight 😅
Opened: https://github.com/projectharmonia/bevy_replicon/pull/181

[echo lion]

Thanks looks good, I’m not my computer to approve it you can merge

[spring raptor]

Going to sleep right now, will finish dont_replicate tomorrow.

[fast tundra]

Is the difference from using bevy_renet is just that this is a more specialized crate?

[echo lion]

bevy_replicon adds automatic replication of server entities to clients (with a bunch of features related to that).

[fast tundra]

Hmm, I see

Also, replicon sounds like a transformer kind haha

[echo lion]

It also enables server <-> client events (bevy events that are sent across the network).

[echo lion]

@spring raptor I made decent progress on the translation layer between visibility components and replicon (after getting sidetracked on a failed mission to implement the macro myself...). There is a fair amount of map accessing required, I am a little nervous about perf: https://github.com/UkoeHB/bevy_replicon_attributes/blob/d79af578daed24598d0dff0733ca13d46d01d62f/src/temp.rs#L20.

[echo lion]

The keys are also heavy at 16 bytes...

[spring raptor]

Renet provides only raw messaging, while replicon provides more high level features, like you see in other game engines.

[spring raptor]

Looks indeed heavy... But you doing many lookups only when you change visibility, so maybe it's fine?

[spring raptor]

What do you mean?

[echo lion]

The hashmap keys are all 16 bytes except entities at 8 bytes. Hashing them is probably more expensive.

[spring raptor]

Oh, I see...

[spring raptor]

@echo lion done: https://github.com/projectharmonia/bevy_replicon/pull/177

[echo lion]

Kind of a mind-bending little topic, but I think it has been solved now 🙂

[spring raptor]

@echo lion do you want me to wait with the new release until you finish with attributes to make sure that the required API is in place?

[echo lion]

Yes please 🙂 it should only be a few days (tm)

I'm pretty excited about the attributes API 🙂 you can even compose visibility settings (e.g. take the Visibilty of two entities and then do and(a, b) to get another Visibility).

[echo lion]

Do you want to do the ClientCache PR now?

[spring raptor]

Forgot about it, will do now

[spring raptor]

Done: https://github.com/projectharmonia/bevy_replicon/pull/182

[echo lion]

Everything is not yet implemented, but I documented the eventual API here: https://github.com/UkoeHB/bevy_replicon_attributes. Still have not worked out the events API.

[spring raptor]

AttributesRepairPlugin will be a part of bevy_replicon_attributes?

[echo lion]

I think so, although I haven't totally decided yet. Need to implement it first then decide.

[spring raptor]

I would more expect it to be a part of bevy_replicon_repair under a feature

Love the API, btw

vis! is nice, I like it.

replicate_to! it confused me at first... I would suggest to show "basic example" with vis! first.

I also think that replicate_to! doesn't shorten match 😅

[echo lion]

It's easier to type lol

[echo lion]

What do you think about renaming Replication to Replicated? It's a big API change but would be a nice long-term quality improvement.

Could even do a deprecation process for the old name.

[spring raptor]

Probably a good idea...

I would go with a straight rename since we are at 0.x version. It's an easy autorename for users.

[spring raptor]

I wouldn't say that it justifies the macro, but it's up to you :)

[echo lion]

Let’s do it in a new release to avoid overloading users.

[crisp stump]

will bevy replicon attributes and repair be eventually upstreamed or will they remain as seperate crates?

[echo lion]

Current plan is to keep as separate crates, although we are likely to put them all in a shared github org

[crisp stump]

cool 👍

[echo lion]

I found a use for ClientCache::get_client_mut() :). It is needed in the visibility cache in case the repair plugin is used - it will try to set the visibility of clients even if they are disconnected.

[candid eagle]

is it possible to procedually register all the client and server events from a macro?

i have way too many modules to keep track of the order in which events are registered on the client vs the server
my current plan is to make a plugin dedicated to registering all events but i would prefer it if i didn't have to manually update its build function every time i add a new event type

I will see if I can find a way to do it and let you know

[echo lion]

I think we can solve this by storing them in an ordered list of typeids, then assigning channel ids based on index.

[candid eagle]

https://github.com/dtolnay/linkme

[echo lion]

Keep in mind the max number of renet channels is 256

[candid eagle]

I believe I can achieve it with this crate

[echo lion]

If you need more than that then maybe a different solution is needed

[candid eagle]

I can always condense some of my related events into enums

[echo lion]

Opened an issue to solve it: https://github.com/projectharmonia/bevy_replicon/issues/183

[candid eagle]

I'll try to sort out an implementation of it in my game and if it works I'll open a PR for bevy_replicon

In theory it would be zero_cost since it gets handled at compile/link time

[echo lion]

Sounds good 🙂

[echo lion]

@spring raptor I decided to add reconnect handling as a ReconnectPolicy config in the VisibilityAttributesPlugin. Repair requires access to the internal cache, so I figure it makes more sense to handle it in-crate (plus it's quite simple, just a couple small systems).

The plugin is implemented, now I just need to write a lot of tests. My favorite!

[candid eagle]

the best implementation i'm aware of atm uses linkme and a wrapper Derive macro for Event + Serialize + Deserialize to populate a distributed slice with the events at compile time, which can then be iterated over from build() on a plugin

currently client and server events are handled by separate (nearly identical) macros because i'm not entirely sure how to parse arguments in a derive macro

[candid eagle]

it looks like my implementation only works if the distributed slice is declared upstream from the events

[spring raptor]

About ordering - it's a totally reasonable request.
It should be easy to solve, just need to store ordered list of event IDs.

I currently at work, but if you want it now - feel free to open a PR.

[spring raptor]

Using derive macro to register events is a bit unusual, it's not how it works in Bevy...
But you can create a separate plugin that adds this functionality, I will mention it in the readme.

[candid eagle]

it already is a separate plugin, the derive macro just uses the linkme crate to add the event to a list which gets added by the plugin

[spring raptor]

Feel free to publish it, I will mention the plugin in the README.
I think it's good to have optional things as separate crates maintained by people who use these features.

[candid eagle]

yeah, it doesn't conform particularly well to the design patterns in replicon or bevy in general, I've also got a lighter implementation that doesn't involve custom macros

[spring raptor]

You mean a version that works like App::add_something and automatically sorts?

I going to make event-registration order independent, I just currently at work right now :)

[candid eagle]

nah, its just what the macro does but implemented manually in a copy-pastable fashion

impl MyClientEvent {
    #[linkme::distributed_slice(CLIENT_EVENTS)]
    fn register(app: &mut App) -> &mut App {
        app.add_client_event::<Self>(EventType::Ordered)     
    }
}

and then this to actually add the events

fn build(&self, app: &mut App) {
  //Register each client event
  CLIENT_EVENTS.iter().for_each(| register | { register(app) });

  //Register each server event
  SERVER_EVENTS.iter().for_each(| register | { register(app) });
}

linkme is maintained by the same core dev as serde, syn, quote etc (at least based on repo ownership)
CLIENT_EVENTS and SERVER_EVENTS are statics that get populated by linkme at compile time with a reference to every register() as a way to guaruntee uniform loading order on client and server whilst still separating the client and server code nicely

the loading order will soon be irrelevant but i still find it helpful to keep code contained, once i've finished making the macro version as ergonomic as possible i'll publish it as a crate and let you know :)

[spring raptor]

TypeId is not stable, I think we need something else.

[echo lion]

Small thought in prioritization: since we set visibility per-entity, we can also set the priority at the same time

[echo lion]

Ah right, might have different sorting between binaries (technically although unlikely)

[spring raptor]

What do you mean?

[spring raptor]

The documentation says about different ordering between Rust releases.

And different hashes.

[echo lion]

When setting visibility to true, also specify a priority (whitelist only). Then use priority as a filter to control replication frequency (you lose global init synchronization).

[spring raptor]

Hm... But it will result in partial world replication...

[echo lion]

That’s what prioritization does lol, it’s opt-in

[spring raptor]

Yes, but could partial replication of visible part be a problem?

[echo lion]

Tbh I don’t know. Maybe it’s game-specific?

Entities with the same priority will synchronize

[spring raptor]

For visibility you directly control what client can and can't see.
But for this one you could receive different visible part based on how good your connection. Isn't this something that we tried to avoid?

[echo lion]

That’s true, and you can definitely use visibility attributes to control the range of visible entities.

Priority would perhaps make it easier to automatically calculate update frequency instead of needing a bespoke algorithm

You could attach priority to client attributes and entity vis conditions, then calculate per-entity priority in the backend?

Although it could result in magical behavior that’s hard to debug (for complex vis conditions)

[spring raptor]

Don't get me wrong, I'm not against prioritization, we just need to decide on how it will work.
If we decide that we allow partial replication, then we don't need to wait for init tick for our updates.

I mean instead of waiting for the world tick, make it per-entity.

I.e. don't apply an update if minimal required tick for an entity is missing.

[echo lion]

You mean on the client?

[spring raptor]

Yes

I understand that prioritization is server thing, but it affects how client apply updates.

But partial replication (of what client can see) could cause weird bugs on shitty connections.

[echo lion]

Hmm I think if an entity is spawned or component added then we always send init message. We only throttle the change detection

[spring raptor]

Oh

Then never mind

It makes sense 😅

[echo lion]

So users would still need an algorithm for adjusting client location attributes as they move around (to control the outer spawn-detection edge), but within visible locations they have priority gradients. Although that implies adjusting the gradients constantly based on relative position with the client so idk.

I guess you’d iterate client attributes after they have moved a certain distance, to adjust location priorities. That means some heavy slamming on the visibility cache esp with lots of entities.

Afaik most games calculate priority right at the replication stage, skipping a lot of the intermediate mapping costs

[spring raptor]

Do you need to iterate over attributes? You only need to know if an entity is visible or not and then update it's priority. Maybe it will be faster to just iterate over visible entities using replicon's core API?

[echo lion]

You’d iterate attributes if using entity vis conditions to compute an aggregate priority.

Also what if you need to make some entities visible? It’s not just about turning off visibility.

[spring raptor]

Why iteration over vis conditions are needed for priority calculation?
I imagined that a plugin that sets priority based on a position should just iterate over visible entities and set priority based on the position.
Or you wanted some more complex priority caclulation based on attributes?

[echo lion]

Priority calculations can include more than just position (at least a full-fledged generic priority system could).

[spring raptor]

Got it. So you think that you need to design attributes to be faster?

[echo lion]

Yeah maybe, I am thinking about the requirements

[spring raptor]

Have you considered doing what physics usually does - a fixed bitflags instead of ids? I guess that this will remove some functionality, but will also make it faster.

[echo lion]

Not sure what you mean, where would I put the bitflags?

I think prioritization with vis conditions is just inevitably slow (slow as in multiple map accesses to update a priority).

I’d rather have a nice API that’s slow (and can maybe be optimized eventually) than a poor API that’s fast (and the API can’t improve).

[spring raptor]

I.e. rooms (like collision groups work).

[spring raptor]

It's up to you and even depends on the game.

So feel free to experiment :)

[echo lion]

Yeah I don’t think my game even needs it. It’s mostly for open-world stuff afaik.

[spring raptor]

Then it's okay to focus on the API niceties.
It's important that the core is as fast as possible.

[echo lion]

Right, core should be fairly easy to do fast.

[spring raptor]

@echo lion love the proposed solution.
Does it covers https://github.com/projectharmonia/bevy_replicon/issues/47 or it's a separate thing?

[echo lion]

Hmm

Maybe?

[spring raptor]

I think that the latest issue #184 superior to this one.

Maybe close older one?

[echo lion]

Maybe we can include a 'priority modifier' based on time since last ack + time since a component was changed.

To lower the priority if we already sent out updates for a component change...

[spring raptor]

Sounds reasonable. But I would research how it's done in other engines first. It's always good to check the "prior art".

But I currently planning to spend my free time more on the game, so I would ask you to look into it if your game needs this :)

[echo lion]

Yeah I also do not plan to work on priority. I like writing down ideas for later 🙂

[echo lion]

@spring raptor for events: ServerEventSender<Type>(&attributes, event, vis!(..)). This is a system param that wraps an EventWriter<ToClients<Type>>. We do this instead of a different event writer in order to synchronize event sending with client attributes (which can change throughout a tick).

[spring raptor]

Ah, right, good call!

[echo lion]

One disadvantage is only Clone events can be sent, because we need to use SendMode::Direct.

Like this: https://github.com/UkoeHB/bevy_replicon_attributes/blob/20b3ef9636c7e15fb8086bdd16e0437a4ac37849/src/server_event_sender.rs#L20

It works fine for me since in girk I pre-pack the message into a Bytes instance, but for general users it's not ideal.

[spring raptor]

Right, but look like it's unavolidable :(

[echo lion]

What if we send to renet directly in ServerEventSender?

[spring raptor]

Hm... This does make sense...

[echo lion]

So include an app extension that registers dummy sending systems with replicon, and stores a serializer function in an Arc in a resource that's accessed by the sender param.

But then you lose the versatility of a system... Maybe require exclusive access then run a system callback? Hmm

I'll stick with the Clone bound for now.

[echo lion]

Realized this today.. one big advantage of having multiple crates (e.g. replicon + replicon attributes) is you can control tracing levels with a lot more precision. Replicon attributes adds a bunch of verbose trace-level logs that you might want to filter out from replicon trace-level logs.

[echo lion]

Progress today: added any!(), all!(), none!() helper macros, added ServerEventSender, and started writing tests. The API should be ~done now, I just need to write a lot of tests.

[spring raptor]

Yes, this is nice!

[echo lion]

@spring raptor ok testing done 🙂 I'm ready for a release. Do you want to do a joint-release to the crates channel?

[spring raptor]

I do!
Would you like to open a PR to the README first to display the mention on crates.io as well?

[echo lion]

done

[spring raptor]

Merged. Do you want me to publish the release announcement first and then you post your announcement right away or you want to combine them?

[echo lion]

Let's combine them. Once you release replicon I can update my crate dependencies and release.

Can you do a brief review of the attributes API first so we can avoid churn?

[spring raptor]

Okay, looking into it now

[echo lion]

One thing I'm not 100% on is vis!() being globally visible. Maybe the crate should ship a special Global visibility condition.

The problem with vis!() is you can silently make an entity global with visibility.remove(x).

[spring raptor]

Nice, I love it! Looks quite convenient. Some notes:

1. I like the getting started guide in the README. I used to do this too, but when I had a lot of types, it looked ugly because they were rendered as [`Type`], but now GitHub has changed the rendering and it looks great. Maybe I should do the same for replicon?
2. Still think that replicate_to! looks a bit too much, but it's up to you 😅
3. ServerEventSender is a bit generic, not quite clear that it's for attributes. Maybe okay, but I would name it something like AttributesEventSender?

[spring raptor]

What if make it depend on Whitelist / Blacklist?

[echo lion]

Github doesn't do [`Type`] if it's a link. All my references are crate links for this reason.

[spring raptor]

Ah, that's why it looks nice.
Maybe worth doing the same for replicon? Or keep it as is?

[echo lion]

Re: ServerEventSender, if you are using this then it's probably the only way you are sending events in your codebase so there shouldn't be any confusion.

[spring raptor]

Yes, just a bit harder to remember

[echo lion]

I think a getting started link is just as good as code in the readme. Replicon has a longer getting started page anyway.

[spring raptor]

Okay!

[spring raptor]

Maybe it's just me

Anyway, I the API is really good

[echo lion]

I will think about the empty vis!() a little bit more. And remove replicate_to!(), it's good to simplify things.

Can you do a replicon release so I can update deps? I did not find any API changes needed for the attributes library.

[spring raptor]

About the announcement - when you will be ready, maybe you will publish the joint-release this time?

[spring raptor]

Okay

[echo lion]

Hopefully within a couple hours

If you want to draft the release for replicon, I will add in details for attributes

[spring raptor]

No rush, I just suggesting that this time I publish it only on crates.io first and when you finish with yours, you will publish the joint-announcement in Discord yourself.

[spring raptor]

Published.

[echo lion]

For efficiency and complexity reasons, the attributes crate does not work with the Blacklist policy (at least as far as I can tell, switching between whitelist and blacklist analysis is mind-bending). Instead of vis!() being global I will ship Global and Client(client_id) built-in attributes, and add those to new clients automatically when they connect.

[spring raptor]

Got it!
Makes sense then.

[echo lion]

Ah I’m not at my computer to check, is it possible to query the replicon visibility policy? It would be good to assert not blacklist in the attributes plugin as a sanity check.

Can add it in a follow-up release too, not urgent

[spring raptor]

Not right now, this one need to be exported: https://github.com/lifescapegame/bevy_replicon/blob/48854217cfa02f45d55c6f747ede6c6d51c777e6/src/server/client_cache.rs?plain=1#L19

Feel free to open a PR, I will draft a new release, releases are not a problem for me.

[echo lion]

done

[spring raptor]

Drafted a new minor release

You will need to use = 0.21.1 (inclide this minor).

[echo lion]

@spring raptor it's great to have this done finally 🙂 visibility control has been on my todo list for half a year

Thanks a lot for your help, it would not have turned out so well on my own

[spring raptor]

@echo lion I recently started a Mastodon account, I’ll post information about the release there.
Do you use it? I would have tagged you :)

[echo lion]

Don't have it, not even sure what it is lol

ah twitter-like

[spring raptor]

It's an open source alternative to Twitter. A microblog basically.
Bevy devs are there.

[echo lion]

I avoid social media like the plague :p

[spring raptor]

I usually too, but started using for the future promotion of my game and increasing motivation (I like to have feedback 😅 ).

[echo lion]

It's a good idea 🙂

[waxen barn]

Is Mastodon more active for Bevy than Twitter?

[spring raptor]

I think so. @cart, @alice_i_cecile and @francoismockers post only there.

[waxen barn]

nice, ill make a mastodon acc

ty

[spring raptor]

I also like that I can cross-post it to Lemmy (Reddit alternative).
Just include @COMMUNITY_NAME@LEMMY_SERVER into the message and it will be visible there. So convenient.

[echo lion]

@spring raptor let me double-check that all my crates compile correctly before merging the PR.

[spring raptor]

Okay, I approved, going to sleep right now, feel free to merge :)

[echo lion]

Yes please on the release :)

[echo lion]

I have now fully integrated bevy_replicon_attributes into my crates, woohoo!

[spring raptor]

Done

[echo lion]

bevy_replicon_attributes is now at v0.2

[runic nest]

I just saw that bevy_replicon can handle server and client in the one App

Support for client and server both in one App and in separate.

Can I write a webserver where player will be able to create a lobby and became a host. And then other players will join
Lets say that our game has two states, Client-only and Client-server (is this possible for bevy_replicon to work as server and also show the game?!).
When someone creates a lobby he became a Client-server, and then other players connects to that one players as Client-only.

Is this a great idea and is this even possible?

[spring raptor]

Answered in #networking . But yes, better ask replicon-specific questions here.

[runic nest]

okay

[plush sparrow]

I was using bevy_qinnet for a long time, but am now eager to try out bevy_replicon - one question though: I have "chunked 'infinite' terrain" and I don't want to send all of the chunks to all of the clients, just the ones around them. Can I manually trigger sending information over bevy_renet? (with just relying on bevy_replicon - I would like to avoid having to pull in bevy_renet myself and keep the dependencies in sync and having to manually enable/disable plugins to get access etc [maybe that's not even needed, it's just something I want to avoid])

I guess https://docs.rs/bevy_replicon/latest/bevy_replicon/#network-events is what I want... but if not "All Renet API is re-exported from this plugin" so forget my question and thanks for the awesome work @spring raptor ! 🙂

[echo lion]

Replicon has client visibility, which is what you want. There is also bevy_replicon_attributes which greatly improves the ergonomics (with some perf cost).

[spring raptor]

In theory you can use events to send chunks manually, but in practice you probably don't want this.

Instead use ClientCache resource to obtain ClientState for specific client and modify ClientVisibility to set visibility per entity.
I assume that your chunks are entities, so just create a system that updates which tiles are visible for each client using ClientVisibility. And replicon will take care of the rest.

Or use the mentioned bevy_replicon_attributes for a more high level API.

@echo lion maybe add a note about this in quick start guide for discoverability?

[plush sparrow]

Thanks, I'll have a look.

[spring raptor]

Pushed some improvements to the quick start guide:
https://github.com/projectharmonia/bevy_replicon/pull/192

I will add visibility example tomorrow

[echo lion]

Ok I will review when you change it from draft mode.

[spring raptor]

@echo lion lightyear's author considering splitting his transport library into separate crate. Do you think it worth to try switching to the new transport lib or stick with renet unless it completely abandoned?
Or just wait and see how the situation develops?

[echo lion]

Id just wait and see. One consideration with a new lib is the need to validate a totally new implementation of netcode.

[spring raptor]

@viscid jacinth we probably wait then ^

[runic nest]

Is there a way to use standard Sprite? I see that it doesnt implement Serialize, but maybe there some ways?

[echo lion]

You can use the blueprint pattern, check the replicon docs

[runic nest]

Oh, thanks!

[timber grove]

Bevy replicon has a ton of awesome features that I really want to use, such as bevy event integration and entity/component mapping and replication. I've been messing around with it for the past week and it has some really great use cases, however In my game I want to do some things that I'm not sure are feasible in bevy_replicon and I'm thinking maybe I'll need to go lower level and just build on top of bevy_renet.

The behavior of replication components replicates them onto a new entity if that entity does not exist, this I want to keep. However, it also seems that the component data is synchronized every frame that it exists for also. Since my game will have large hordes of enemies and projectiles, and should allow for a large amount of clients in a single session, I don't think it will be feasible to synchronize these component values every frame. But since my code is largely deterministic, there's no need for them to be synchronized each frame. Is there a way to set up replicon so that the components are only created if they don't exist? and then field synchronization can be called manually for each entity upon request, maybe by inserting another component that marks them for that?

If so, where could I find more information about doing this? I haven't seen anything in the docs but it's possible I wasn't searching for the right keywords or just am completely blind

[echo lion]

One thing you can do is instead of replicating components is replicate an 'instruction' from which component changes are derived. So basically the blueprint pattern (which is in the docs), but also periodically updating the blueprint.

If the instruction changes with less frequency and is smaller than your components, it would reduce network cost.

[timber grove]

wait components are only replicated across the network if they are changed from the previous frame?

[echo lion]

Close, they are sent if changed since the last ack from the client

[timber grove]

ah, right, that makes sense. didn't know this optimization was implemented though

that's a good idea

I guess something that I'm unclear on (since I'm brand new to netcode in general) is, I don't know where I would put that component. I'm guessing I'd need to put it on the component on the server, right? If I put the instruction on an entity only in the client, would it also be replicated to the server?

[echo lion]

You'd add it as a replicated component to the server entity that needs the instruction. The instruction component is then replicated to the client, which should have a system that listens for Added/Changed on that instruction component, and updates the Added/Changed client entity (which will have been automatically spawned for you by replicon if needed) with the relevant components that you actually want (you'll need to do clock sync in order to line up projectile position/velocity with the current estimated position/velocity on the server).

IMO you need to do clock sync even with small numbers of projectiles, because if you try to replicate physics variables directly then you'll get a ton of jitter (which won't be visible on localhost tests which often have zero latency).

[timber grove]

sorry, unfamiliar with the term clock sync, what is that? is it kind of what's described in the docs as "mapping to existing client entities"? I wasn't actually planning on synchronizing projectiles at all between the clients/server. just having each instance of the game individually create and calculate projectile positions/other properties based on the user input.
My idea was that I could have each client do as much work locally as possible and contribute to reducing server overhead, and only synchronize when needed. Since it's just coop and not competitive I don't think it has to be 100% accurate. Like I was thinking that each arena session is as deterministic as possible starting with a specific seed which determines what enemies spawn and where, and then each client would calculate what paths those enemies take, what players they target, etc, all as deterministically as possible and then when a player damages an enemy, or is damaged by an enemy, resync that enemy with the server and if there's a conflict, use the server's data and override the client data. All ragdoll and other physics stuff would just be for visual umph and not have any gameplay or two-way interactions with the player so it shouldn't need to be synced

[echo lion]

By clock sync I mean matching the server's clock with the client's clock so you can map server timestamps (e.g. 'when a bullet was spawned') to the client ('bullet spawn server' -> 'bullet spawn client' -> + velocity * (current client time - client spawn time))

You can probably get away with assuming server/client clocks are the same unless doing a really hardcore game

[timber grove]

ah yeah, ok. that makes sense

[echo lion]

So clock sync isn't necessary, but what is necessary is projecting spawn time to current time.

[timber grove]

right, kind of similar to rollback, so that the missing time is resimulated for the projectile

[echo lion]

So for this design I would consider a two-part solution. One is the 'deterministic instruction' that is replicated when instructions are updated. The other is 'save point' where you save the entity's current state periodically and replicate that in order to fix desyncs/conflicts (instead of doing rollback).

Basically just paraphrasing what you said lol

[timber grove]

no this great, thanks for talking with me about it. Definitely helps! It's reassuring that it sounds like this approach will be feasible to do in replicon. I really didn't want to have to go down to bevy_renet since I'm not at all comfortable with netcode yet 😅

[echo lion]

Happy to help :)

Live-action replication is a real rabbit hole, so finding a workable simplification is quite useful

[timber grove]

I guess what I was asking here was, for the client, for example if I have the player of the current client attack an enemy, could I just attach an Instruction component on the enemy entity on that client? or would I have to send a message or event to the server and then the server would attach the instruction to that enemy?

like, is the server aware of replicated components on client entities?

[echo lion]

Yeah the client does not replicate anything to the server, you need to send an event to the server.

[timber grove]

okay that makes sense thanks

[spring raptor]

@obsidian parcel

How does it know which components/entities are which across the network? I assume it does some sort of internal tracking
Yep, when a client receives replication message, it automatically maps all entities (and entities inside components) into local ones using this resource:
https://docs.rs/bevy_replicon/latest/bevy_replicon/client/client_mapper/struct.ServerEntityMap.html

but I noticed you need to spawn the entity on both the server and the clients and then once spawned it keeps the components in sync
This is not how it works. You need to spawn an entity only on server and it will be automatically spawned on clients.

However, if you want to spawn something on client ahead of time and notify server about it, you can use this resource:
https://docs.rs/bevy_replicon/latest/bevy_replicon/client/client_mapper/struct.ServerEntityMap.html
Docs also contain an example.

[obsidian parcel]

I see that might explain why the components on the client don't seem to be updating for me. I did see this:
https://docs.rs/bevy_replicon/latest/bevy_replicon/index.html#blueprints-pattern
I think because my client entities need components that can't be serialized I'll have to use this pattern? For example most of the sprite bundle can't be serialized.

[obsidian parcel]

hmm I can't seem to get the replicate features to work correctly. 😦

I know the client/server are connected, I can also see the entity being spawned on the server

but the client never gets the data.

nvm

I fixed it 😄

I got a little over zealous removing server components

It's nice you can implement your own serialization/deserialization as well.

I only really need to send the translation with the transform.

save on a ton of bytes 😄

[spring raptor]

Yep!
It would be also wastefull to send all such data to clients.
And the idea is to just send your custom component (like ImagePath) and create a system which inserts the image bundle.

[obsidian parcel]

yeah I'm only sending stuff that needs to get "replicated" Thankfully everything is fairly hard coded in terms of assets so I don't need to send strings.

[spring raptor]

Then it's even better!

[spring raptor]

I wish bevy turn Transform into a bundle and implement querying for bundles, but that's for the future.

[obsidian parcel]

yeah bundle queries would be nice!

Well networking seems "easy" so far, I think it just takes a bit of a shift in thought with what data should the server hold vs client and what events etc happen. Eventually visiblity is going to be a big pain, but for now I'll just send everything.

[tardy knoll]

Hi everyone - hopefully it's the correct place to post.
Giving bevy_replicon a try, I would like to understand what explains the jankiness of the client movement in this video below. Sending events over an UDP socket on the same host should be pretty quick, no? Morever, even when interacting with the client it looks better on the server.

Thank you!

[spring raptor]

If you find the built-in visibility system too barebones (https://docs.rs/bevy_replicon/latest/bevy_replicon/index.html#client-visibility), you can take a look at this crate https://github.com/UkoeHB/bevy_replicon_attributes

[obsidian parcel]

This looks normal to me, there is still overhead from the networking and to solve this I think you'd need interpolation between packets.

you might be able to increase the server tick rate?

[tardy knoll]

Thank you, I realize now that it's configurable; this made the difference

[spring raptor]

That's because we don't send updates every frame by default, see https://docs.rs/bevy_replicon/latest/bevy_replicon/server/enum.TickPolicy.html
By default it’s 30 ticks per second.

To avoid jittering you need to interpolate. So you may want to take a look at https://github.com/Bendzae/bevy_replicon_snap
But it also depends on what kind of game you making. If it's a fast paced game, you may also need prediction, for this you may take a look at proof of concept integration with bevy_timewarp: https://github.com/RJ/bevy_timewarp/blob/main/REPLICON_INTEGRATION.md
I mentioned it in the example in comment.

My goal was to make a modular core crate. For example, my game don't need interpolation and prediction. But some games may need.

[obsidian parcel]

keep in mind in production(across the internet) you'll be increasing the number of packets sent its better to interpolate where possible I think. I'm still new to this networking stuff though

[tardy knoll]

Thank you - I'm trying to make a vampire survivors-like multiplayer game (the billionth game on the market ofc)
I also gave timewarp a try, but I couldn't run the example; I tweaked the code to compile, but something was missing:

Requested resource bevy_replicon::replicon_core::replication_rules::ReplicationRules does not exist in the `World`.
                Did you forget to add it using `app.insert_resource` / `app.init_resource`?
                Resources are also implicitly added via `app.add_event`,
                and can be added by plugins.

I will give it another go now that I figured out the basic example with no interpolation/prediction

[spring raptor]

The example in bevy_timewarp is a little outdated. I would appretiate if someone make an integration-level crate to automate it. Something like bevy_replicon_timewarp.

The error means that you didn't initialize replicon plugins. ReplicationRules is a resource from RepliconCorePlugin from ReplicationPlugins group. Make sure that the plugin is enabled.

[tardy knoll]

Oh, woops, it's actually bevy_replicon_snap I was checking out:
https://github.com/Bendzae/bevy_replicon_snap/blob/main/examples/interpolated.rs

There's no example for bevy_timewarp, no? At least I don't see one

[spring raptor]

I assumed that you mean the code from the REPLICON_INTEGRATION.md from bevy_timewarp.

Let me take a look.

[spring raptor]

Will do in a moment, need to finish the review of 0.13 update PR first.

[echo lion]

Check out bevy_replicon_attributes, it is designed to make visibility as easy as humanly possible. Oh Shatur linked it lol

[obsidian parcel]

yep, I will once I get around to it. I don't even have fog of war stuff "yet" so I'm not super worried.

[spring raptor]

The example works fine for me.
But looks like the replicon version in the example is a little bit outdated:
https://github.com/Bendzae/bevy_replicon_snap/blob/cd3e6c4390853fa00dbb3d1c925aeff128ac62ba/Cargo.toml#L12
You probably copied the code into your project and have this issue. Try setting your version to "0.17" to make them match and it should work.
If you planning to use this crate - try updating it to the latest version and send a PR to the author. It shouldn't be hard. Feel free to ask questions about the migration. We also have a changelog to make this process easier: https://github.com/projectharmonia/bevy_replicon/blob/master/CHANGELOG.md

[obsidian parcel]

@spring raptor is there a way of mapping entities for events?

[spring raptor]

Yep, just like components:

1. Implement https://docs.rs/bevy_replicon/latest/bevy_replicon/replicon_core/replication_rules/trait.MapNetworkEntities.html
2. Use special _mapped function to register event / component.

[spring raptor]

After 0.13 we will use the built-in trait for it: https://docs.rs/bevy/latest/bevy/ecs/entity/trait.MapEntities.html

[obsidian parcel]

Thank you, I was looking for add_client_event_mapped 😅

Works great

[timber grove]

I don't suppose there's a way to replicate Resources in replicon? The same way that components are

[echo lion]

Not currently, no

[timber grove]

bummer, would be a nice feature

[spring raptor]

Agree, it would be nice to have. But for now you can use events to sync resources manually. Or you can turn them into entities.

[timber grove]

I can't seem to figure out how to send events from a server to client? I have the event registered with app.add_server_event::<ClientConnected>() and I have these two systems to send and receive the event, the event gets sent from the server, but the event never seems to get received, and the event message gets logged on the client.

am I doing something wrong?

I couldn't find any examples or explanations in the docs on how to send/receive server events, so this is kind of just my best guess of how to do it

[spring raptor]

Have you read the docs?
Because we have the example right in the documentation:
https://docs.rs/bevy_replicon/latest/bevy_replicon/#from-server-to-client
If not, I would highly recommend to read the entire quick start guide.

[timber grove]

apparently not 🤔

[spring raptor]

Remove ToClients wrapper.

[timber grove]

didn't seem to show up in the search for some reason

[spring raptor]

On client you read/write all events without any wrappers, just like usual.

[spring raptor]

It's on the main page.

[timber grove]

well I see it now that you linked it haha, thanks

[spring raptor]

Maybe I should split it to put it into their function...

[spring raptor]

And on server you use wrappers FromClient / ToClients because we need additional information (who the message came from and who to send it to).

I tried my best to write the docs, but I'm not a native speaker 😅 And my native language differs a lot from English.
So if you find something confusing - I would highly appreciate a PR.

[timber grove]

I appreciate the help, thanks. I'll give the quick start guide a more thorough read through and open a PR if anything is unclear. But from what I've read so far, it seems very clear and well written. I guess I must have just missed that example because it wasn't what I was looking for when I glanced over it the first time.

[spring raptor]

Feel free to ask any questions

[spring raptor]

I have an idea. We probably need to put a link to corresponding quick start guide sections in add_server(_mapped)_event / add_client(_mapped)_event.

[timber grove]

that would definitely be helpful

[iron flare]

is there any pr for replicon that currently supports bevy 0.13?

[spring raptor]

Yes:
https://github.com/projectharmonia/bevy_replicon/pull/198

[spring raptor]

@echo lion thinking about https://github.com/projectharmonia/bevy_replicon/issues/27
I think I know how we can achieve it with low effort and without generics.

1. Introduce ServerState and ClientState (or maybe RepliconServer and ReplicionClinet?) structs. These structs will store messages in form of bytes, some utility fields (like bool if the client connected), but no I/O.
2. Introduce our own common conditions (connected, connecting, etc.), connection events, ClientId, send-type enum, etc.
3. Require backend crates to drain messages from structs from 1 and update utility fields.
4. And for renet we provide bevy_replicon_renet that works like bevy_renet, but also does 3. It will be a very small crate, 4 systems (sending and receiving for client and server). And writing backends will be extremely easy.

This way other backends can provide something similar to bevy_replicon_renet (like bevy_replicon_aeronet or bevy_replicon_xwt, anything). We will maintain only renet, other backends will be up to others.
I think it's good, any alternatives to renet will have a high-level networking almost for free.
Our unit test, integration tests and benchmarks will be able to pass messages to each other simply by draining, no I/O. We also will be able to simulate packet loss.
And as a small bonus - we won't need to depend on bevy_renet, we can use just renet.

What do you think?

[echo lion]

I think it is feasible, although I am uncertain about having our own connection events, since those can vary widely depending on the backend.

[spring raptor]

Ah, right. Then we can require backend integration to provide one more system which updates connected clients list.

[spring raptor]

@raw idol will you be interested in something like this?
And maybe @drowsy lance. I know that the issue is quite old, but we were focused on different things. Are you still interested?

[drowsy lance]

Yep

But also: https://github.com/lucaspoffo/renet/issues/148

Just created it recently

[raw idol]

I like this idea, it makes sense and sounds pretty modular

altho I'd also be worried about how replicon is tied to renet channels

[spring raptor]

Renet already support pluggable transports, there are PRs for memory channel (https://github.com/lucaspoffo/renet/pull/117) and WebTransport (https://github.com/lucaspoffo/renet/pull/107).
Last time I talked with the author, he said that he prefers them to be in separate crates. So you can create your own crate right now and maintain your transport there.

[raw idol]

I'm not a big fan of how renet handles multiple transports. It still feels like the core code is tied to netcode, and SocketAddrs and so on, and that additional transports are more of an extra rather than being considered from the beginning

[spring raptor]

We only need the ability to reserve specific channels based on events. And backend integration will need to provide NetworkChannels conversion into backend channels.

Shouldn't be hard too.

[raw idol]

are the channel IDs linearly increasing?

0, 1, 2, etc.

[spring raptor]

Right now - yes. Is this a problem?

[raw idol]

no, that's perfect

since I use a varint encoding for the lane ID

and any reasonable transport should do so too (in my opinion 😉 )

[spring raptor]

Great!
So if you like this idea - I will take a look into it soon and provide a proof-of-concept PR to evaluate.
Just want to finish door system for my game first, shouldn't take a lot of time. This will also give renet's author time to draft a new release. If he will draft a new release - I will also draft a new one with the current state and decoupling will be a part of the next one. Will be easier to migrate for users, 0.13 contains a lot of changes alone, it's hard to deal with all of them at once.

[spring raptor]

@echo lion could you take a look at https://github.com/projectharmonia/bevy_replicon/pull/201 ?

And I will release.

[echo lion]

Another small issue with this is it's useful to have the RenetReceive/RenetSend system sets in order to precisely insert systems between renet and replicon. I do this in bevy_replicon_repair in a couple places (it is not necessary to have the renet sets, but it's nice to have them).

[spring raptor]

We re-export everything from renet.

[echo lion]

I wouldn't call that a blocker, but something to consider.

[spring raptor]

So I'm not sure what you mean

Could you elaborate?

[echo lion]

I'm talking about the 'more modularity' proposal where renet would become invisible to the base replicon crate.

[spring raptor]

Ah, sorry, missed that you replied to that message.
Agree, we probably need to provide sets like this.

[echo lion]

Maybe like a ClientSet::PreSend?

[spring raptor]

How about ClientSet::TransportSend?..

Or it could be technically not a transport...

ClientSet::BackendSend?

I want to make it obvious that backend providers should put their systems there.

[echo lion]

ClientSet::SendPackets

And ClientSet::ReceivePackets

[spring raptor]

Love it

If you have free and want to colaborate, you can start the separation.
I will be able to join soon, just need to finish stuff for my game. I implemented the mentioned door system, but I also want to update to 0.13.

[echo lion]

I am a little swamped right now so it will be a little while before I can dive into it.

[spring raptor]

No problem, I will start myself then. I think it will be on this week.

[spring raptor]

Oh, almost forgot about one tiny issue with replicate_into. It doesn't update existing entities in the scene.
Will fix and then release.

[spring raptor]

@echo lion Okay, now hopefully the last one: https://github.com/projectharmonia/bevy_replicon/pull/202

[echo lion]

Done

[spring raptor]

Released!
After I finish updating to 0.13, I will take a look into I/O abstraction.

Bevy definitely increases the development velocity and it's good. But if you have a relatively large game, migrations are pain even with their migration guides 😢

[spring raptor]

Finished the update, looking into I/O abstraction now.

[spring raptor]

@echo lion opened a small PR with minor refactoring: https://github.com/projectharmonia/bevy_replicon/pull/203
It would be great to merge it before the abstraction.

[echo lion]

Done

[spring raptor]

Thanks!

I/O removal is on the way, it already compiles without renet, but I need to fix tests, benchmarks and write the bevy_replicon_renet.

[echo lion]

Sweet thanks for taking charge :). I am now in the weeds studying mod_picking to hopefully start upstreaming picking support to Bevy.

[spring raptor]

In Renet disconnect event contains disconnect reason error enum.
Should I provide something like this?

I just not sure if I can make it generic enough...

[echo lion]

You can Box<dyn std::Error> I think. It’s important to be able to log messages emitted by the transport.

[spring raptor]

Okay, all tests pass and all benchmarks work.
Writing the renet plugin now.

[spring raptor]

Looks like I can't derive Event then:

error[E0277]: `(dyn StdError + 'static)` cannot be sent between threads safely
   --> src/server/replicon_server.rs:164:10
    |
164 | #[derive(Event, Clone)]
    |          ^^^^^ `(dyn StdError + 'static)` cannot be sent between threads safely
    |
    = help: the trait `Send` is not implemented for `(dyn StdError + 'static)`
    = note: required for `Unique<(dyn StdError + 'static)>` to implement `Send`

[echo lion]

Add Send/Sync bounds to the trait object.

[spring raptor]

Do you think that most errors have it?

[echo lion]

Yeah

It's only going to not be send/sync if there is a raw pointer, most of the time

[sharp roost]

Almost all types have send+sync

[spring raptor]

Got it, thanks!

@fluid moth I saw your message and yes, I suspect that it's pings from Renet...

Could you try to run any renet example to confirm?

[fluid moth]

Hi! I am suffering from a flood of small packages from the bevy_replicon. I've disabled replication of all the components and events and still see 3000-7000 small(70 bytes) packets per second. which ends up to be quite a lot. The official tic tac toe example sends 120 packets per second regardless of the tick rate set up in the bevy_replicon plugin which makes me think if this is tied to the framerate (my monitor is 60 herz)

Has anyone run into similar issues?

[echo lion]

What is your FPS?

[fluid moth]

I run with "minimal plugins" on my project, let me quickly see how fast does it spin the update schedule

[spring raptor]

@fluid moth could you try any renet example? It's the messaging crate we use. Here is the repo: https://github.com/lucaspoffo/renet

[sharp roost]

Did you add in the plugin that makes it go at X fps, rather than as fast as possible? I forget what it's called

[fluid moth]

The official plugin sends ~250 packets per second, which is weird

[spring raptor]

You mean renet?

[fluid moth]

I'll try that, thanks!

[fluid moth]

yes the demo_bevy from renet repo

[spring raptor]

Thanks, then yes, it's probably keepalives.
Now I have to redirect you to #1038137656107864084 😅

[spring raptor]

Damn, Renet's DisconnectReason doesn't implement Error.
Do you think it's fine if I go with a string?..

[echo lion]

Oh yeah cause it's not an error duh lol. Yeah a string works.

[spring raptor]

I will submit a PR, it should be an Error.

But for now will go with a string then.

[raw idol]

does renet implement its own graceful disconnect system?

or is that part of netcode?

[spring raptor]

It's a mix of both, see https://docs.rs/renet/0.0.15/renet/enum.DisconnectReason.html

[echo lion]

It is not totally graceful, if the client crashes then the server has to wait to time out.

Even if you try to reconnect with a new token, you still need to wait for the old connection to time out.

There is a solution though, that needs to be implemented: allow new tokens to supersede old tokens if the token timestamp is higher (implying the token issuer permits superseding the old token).

[raw idol]

yeah but if the client crashes then there's basically no guarantees of what happens 😉

[spring raptor]

Getting closer, everything works.
Thanks to the huge pile of tests we have I quite confident in changes 😅
Working on docs.

[spring raptor]

@echo lion done: https://github.com/projectharmonia/bevy_replicon/pull/204
Don't be scared by 2k+ diff 😅 I assure you that it's not that hard to review, changes are quite straightforward.

[echo lion]

Big diff lol, this will take a bit

[spring raptor]

Sure, take you time :)
I recommend to start reading from the changelog.

[spring raptor]

@raw idol @drowsy lance you don't need dive into the internals, but you may be interested in how renet integration is done: https://github.com/projectharmonia/bevy_replicon/blob/io-abstraction/bevy_replicon_renet/src/lib.rs
It's only a single file.

[drowsy lance]

Sorry, is this new?

Ooh, nice

Closes https://github.com/projectharmonia/bevy_replicon/issues/27

[spring raptor]

Yes, you said that you are interested in I/O abstraction. Now renet integration is provided by a separate crate.

[drowsy lance]

I think I can integrate xwt now, sweet

I'm actually planning to take my experiments game project and try different networking libraries that run on top of xwt

[spring raptor]

I provided the instuction in the documentation, but it won't be convenient to read it without compiling the branch. This is why I pointed to renet integration file, could be used as a reference.

[drowsy lance]

Thx, I'll fibure it out

[spring raptor]

Great, let me know if you have any questions or suggestions about the API.

[raw idol]

interesting, this looks fairly simple to integrate

how does connection work in this?

I'm seeing a set_active call and some Connecting and stuff, but how does the user actually trigger a connect?

[spring raptor]

User need to create a messaging library-specific server or client

And then the integration layer of the library is responsible for updating the RepliconServer or RepliconClient.

[raw idol]

makes sense

and do you disconnect the client/server through replicon, or through the transport-specific API?

[spring raptor]

Right now - yes.

But I planning to provide events for both sides and hide set_active/set_status.

[raw idol]

is that "yes" for through replicon or through the transport? 😅

[spring raptor]

Ah, sorry, I read the message wrong. Through the transport-specific API.

[spring raptor]

Solved as part of the PR. I found an easy way to do it without events.

[spring raptor]

@echo lion how do you disconnect gracefully with renet?
There are 2 cases:

1. Disconnect by client.
2. Server stop.

Do you issue a disconnect(s) first, wait an update and then remove resources?

[echo lion]

If the client wants to manually disconnect, you call RenetClient::disconnect().

[spring raptor]

So you don't remove the resource?

[echo lion]

There is no need to remove resources (other than annoying transport error spam, which needs to be fixed).

[spring raptor]

Ah, okay, then let's consider transport error spam as renet bug 😅

@echo lion but what about server?

[echo lion]

When reconnecting, you replace the resource on the spot (although you do need to drop the existing NetcodeClientTransport before setting up a new client in order to free the socket address in case it is being reused).

[echo lion]

The server is assumed to last forever.

[spring raptor]

Then I probably drop resources after disconnect signal... Since we need to handle it on our side now.

[echo lion]

There is RenetServer::disconnect_all, which you can call before shutting down.

[echo lion]

I disagree, this forces users to use Res<Option<>> everywhere.

[spring raptor]

Yep, saw it. Will probably issue a disconnect and wait for an update then.

[spring raptor]

Not with the current impl. Users won't need to touch RenetClient after initialization.

RepliconClient will always be present.

[echo lion]

I'll have to review the implementation to see if it works. Not ready to look at it quite yet.

[spring raptor]

Sure, take a look and we will discuss then. I just noticed that the disconnect is not properly handled.
For now I will try to "automatically remove renet resources" approach, because want to minimize the interaction with library-specific code.
But we can reconsider after your review.

[spring raptor]

No, it's too fragile due to how renet works. In order to disconnect and remove resource I need to send a disconnect, wait the whole app update and then remove resources. This could break users workflow. Will keep disconnects as before for now.

Renet registers disconnect in PreUpdate and then send packets about it in PostUpdate. This is why the whole update is needed.

[floral saddle]

In the abstract IO PR, the no_connection method seems like it could have a more descriptive name.

[spring raptor]

Yes, I thought about it too... Do you think that I should keep old has_authority name? Or would you suggest a different one?
Feel free to left any review comments, it always helps.

[floral saddle]

I think has_authority certainly reflects the intention of the caller better, even if it were just a call-through to the new function

[spring raptor]

Thanks for the feedback, pushed the rename into has_authority.

[obsidian parcel]

When a client disconnects and I want to "restart" the server is there a best practice for this? Right now I keep track of all of the entities with a tag and just despawn them recursively which seems to work. I am seeing some issues with client id's when the client attempts to connect again.

[spring raptor]

There are several options, it depends on what you want.
If you want to go to the game menu on disconnect - yes, just despawn all game session related entities. You probably already have a logic for this for you "go to main menu" button.
If you want your game to freeze with a message "attempt to reconnect", you can also do the mentioned re-spawn, but there is also https://github.com/UkoeHB/bevy_replicon_repair to "repair" your state without despawning entities.

About why reconnect client ID issue I not sure. Could you elaborate?

Going to sleep right now, will be able to answer tomorrow.
But maybe @echo lion will be able to answer sooner, I think it's day for him 😄

[echo lion]

Do you mean reconnect the client?

I wrote bevy_replicon_repair to help support client reconnects.

[obsidian parcel]

It appears as though the client ID is new when they connect to the server but then the FromClient says the old original ID. I imagine its doing some sort of client reconnect logic?

[echo lion]

Aside from that, you need to issue a fresh ConnectToken to your disconnected client. With the same ClientId.

Then the disconnected client re-creates the RenetClient and NetcodeClientTransport resources (be sure to drop NetcodeClientTransport before making a new UdpSocket in case you are using the same client address).

[echo lion]

Client ID should not change unless your server creates a new ConnectToken with a different client id.

[obsidian parcel]

ahh, why would I see a new one when the client first reconnects?

[echo lion]

Where do you see a new one?

[obsidian parcel]

in this example the clientID ends in 308 for the first connection

that connection is ended by the client which causes the server to wipe entities and some other data

a new client joins with the client id ending in 757

but then that client sends a message as 308 somehow

I do not understand

My guess is that the server knowns who that client is

and tells them their client id should actually be X

but I'm not sure how I can handle that in my own code.

part of the issue is that I rely on client id to identify clients maybe I can't use that.

[echo lion]

What is your process for generating ConnectTokens and distributing them to clients? Also where are you reading client ids for printing them?

[obsidian parcel]

Client connect

I don't have client tokens yet

[echo lion]

It looks like client ids are being printed from 2 different spots in your code, is there some staleness/disjointedness there?

[obsidian parcel]

One comes from client connect event

The other comes from FromClient

Wrapper around an event

[echo lion]

Are you using some code for setting up clients copy-pasted from somewhere?

Can you make a minimum reproducible example? That would help me track down your issue.

[echo lion]

ClientId definitely can be used to identify clients, it's just a matter of managing their lifecycle correctly.

One thing you could do is find the line in your code where client ids are generated (I'm guessing you are using the system clock?) and then print when that line is called. My guess is you are generating client ids when you don't intend to.

[obsidian parcel]

I'll see if I can create a small demo

[echo lion]

@spring raptor I finished (yet another) little project #showcase message, so I'll prioritize reviewing the PR tomorrow.

[spring raptor]

Not sure how it can be possible, IDs taken from RenetServer:
https://github.com/projectharmonia/bevy_replicon/blob/d4c287610edc91b78ba728eea066b5dc1c77b8d4/src/network_event/client_event.rs#L185

Could you try to print RenetServer::clients_id?

[spring raptor]

Great!

[obsidian parcel]

I was able to fix this by using client ids to filter out events that don't match. I suspect the event writer just contains old events for some reason.

[spring raptor]

That's odd because we cleanup them...
https://github.com/projectharmonia/bevy_replicon/blob/d4c287610edc91b78ba728eea066b5dc1c77b8d4/src/network_event/server_event.rs#L291
https://github.com/projectharmonia/bevy_replicon/blob/d4c287610edc91b78ba728eea066b5dc1c77b8d4/src/network_event/client_event.rs#L240
Could you check if these systems trigger for you?

[spring raptor]

Never mind, I realized that we don't to cleanup Events<FromClient<T>> on server when a client disconnects.
I will push a fix today after work (in a few hours).

Looks like we don't have a method to retain events by specific predicate.
I will report it to Bevy.

[spring raptor]

https://github.com/bevyengine/bevy/issues/12204

[echo lion]

Ah that does remind me I saw something like this in my logs in the past, I though it was maybe a renet bug.

[spring raptor]

I think it would be better to handle on our side, but we need the mentioned retain...

[echo lion]

Ended up working to release bevy_cobweb today, didn't do any review. Hopefully tomorrow!

[spring raptor]

It's okay, we are not in a rush :)

[spring raptor]

The I/O PR has been merged!
Huge thanks to @echo lion for the heroic job of reviewing such a huge diff. He suggested a lot of improvements.

I will open some follow-up PRs today and draft a new release.

[spring raptor]

@echo lion PRs are open :)

[timber grove]

is there a way to get the timestamp or tick that an event was sent from a client with EventReader<FromClient<MyEvent>>?

so that I can adjust for it by simulating however many ticks have passed since then?

Currently I'm just including the timestamp in the event but I'm starting to think this is probably already a feature that's built into the crate already

[echo lion]

Approved

[echo lion]

Replicon does not send timestamps or ticks, other than RepliconTick which is used for acking packets and synchronizing updates (probably not useful outside replicon).

[timber grove]

yeah I was reading about replicon tick which is what lead me to ask this

Why wouldn't it be useful?