bevy_replicon

[dire aurora]

In my code I use the previous frame's ConfirmHistory (stored in a component called PreviousConfirm) and the current ConfirmHistory to create a mask containing only the new bits (if there was no PreviousConfirm the current one is used, because they are all new)

[spring raptor]

Right, so we could let users react using Trigger<OnChange, ConfirmHistory> and get all ticks that were written since the last frame from the component (provide a field and a getter for it).
Or provide a dedicated trigger, like Trigger<HistoryUpdated> with the ticks.

[dire aurora]

Using that mask I then do this to turn it into what I need to roll back to the correct tick:

let first = confirmed.last_tick().get().saturating_sub(64 - mask.leading_zeros());
if first < oldest.get() {
    oldest = RepliconTick::new(first);
}

[spring raptor]

Where mask is PreviousConfirm?

[dire aurora]

If both of them have ComponentIds we could probably have one trigger, and use ConfirmHistory vs HistoryUpdated for the ComponentId, then cases like rollback can respond to either of them

[dire aurora]

No, mask is the differences between PreviousConfirm and ConfirmHistory:

let mask = if let Some(mut prev) = prev {
    prev.change_last_tick(confirmed.last_tick());
    let new = confirmed.mask() & !prev.mask;
    prev.mask = confirmed.mask();
    new
} else {
    commands.entity(e).insert(PreviousConfirm {
        mask: confirmed.mask(),
        last_tick: confirmed.last_tick(),
    });
    confirmed.mask()
};
if mask == 0 {
    continue;
}

change_last_tick just shifts it left so they both end at the same tick

This code is also where I used to do that ugly transmute hack 😅

iirc because mask() wasn't a thing yet

[spring raptor]

Under HistoryUpdated you mean UpdateHistory (the resource)?

[dire aurora]

Ah, yes

And the event or trigger should probably be named something something like OnHistoryReceived rather than OnChange. Then if anyone wants to use it to check if we actually changed the current value (if we even write history masks for that) they can check that the bit for the last tick is set 🤔

[spring raptor]

Agree!
So we need to count the number updates, provide a resource (possibly named UpdateHistory) and an observer OnHistoryReceived for both the resource and the ConfirmHistory component. Sounds very ergonomic.
What do you think, @echo lion?

[dire aurora]

Could be an event too ofc.

I wish we had buffered observers so there would be less performance penalty on observers, cause the other benefits observers offers are pretty big 😅

[spring raptor]

I think with observers it will be more ergonomic and plays nicely with assigned ComponentId. Since the event is quite small, it shouldn't be a big perf issue, right?
I also expect observers to continue to improve.

[dire aurora]

Yea I think the main overhead here is just firing all the registered observer for each trigger. Iirc the overhead of observers isn't super big when none are actually registered

[spring raptor]

What I dislike about triggers is how inconvenient to write tests for trigger-based behavior 😅
You have to write some abstraction that observes and collects the events for you to assert them later.

[dire aurora]

Yea that's true, same issue as testing things like running schedules 😅

I have this one rollback test that not only has some awful app init to write it to a resource, it also has this ugly assertion:

assert_eq!(
    **app.world().resource::<Schedules>(),
    vec![
        // The rollback to tick 14
        PreRollback.intern(),
        Rollback.intern(),
        PostRollback.intern(),
        // Resimulation of tick 14
        PreResimulation.intern(),
        FixedUpdate.intern(),
        PostResimulation.intern(),
        // Resimulation of tick 15
        PreResimulation.intern(),
        FixedUpdate.intern(),
        PostResimulation.intern(),
        // Back to present
        BackToPresent.intern(),
        // The regular fixed update
        FixedUpdate.intern()
    ]
);

All these schedules are how I reduced the plugin configuration of the rollback crate btw. The library user only needs to specify which schedule to roll back (could be FixedUpdate like in this test, or a custom one that is run inside FixedUpdate), and which schedule to store items in (has to be contained in the other schedule, or can be the same one, in which case you need to order things relative to a SystemSet)

[spring raptor]

But also very readable, nice!

[dire aurora]

Only because I added comments and rust analyzer formatted it nicely for once 😅

[spring raptor]

Well, maybe 😅
But only the result is important

[spring raptor]

BTW, you can omit vec!, will look a bit nicer.

[echo lion]

Ok

[dire aurora]

Funnily enough while looking for that test from the thing I sent above I ran into tests for that awful unsafe hack that caused me to want to create one history per type ... I've been working on this one history stuff for so long I almost forgot why I even started it 😅

Oh right @spring raptor I forgot to mention what I even plan to do with the World in the write functions.
My history is HashMap<ComponentId, ComponentHistory>, and to initialize a history I need RollbackRegistry to fetch some function pointers for each type, and RollbackFrames to spawn it at the right capacity. I can't get ComponentId in write so the authoritative history is TypeId (which has some potential flaws if we ever want to deal with dynamic or non-Rust components), and if the history is missing I need to send a Command just so I can fetch those two resources, then duplicate everything from write there, except actually init the history if missing there

If we can access resources and Components I can avoid all of these issues

[spring raptor]

I think we can easily add ComponentId to the ctx.

But in Bevy we can't have a dedicated resource access, right?
Maybe we could provide our obstraction for it instead. Could be better then DeferredWorldExcept 🤔

[dire aurora]

Yea there's no dedicated way to get resource access afaik

But once we get "resources as entities" the issue could come back again even if we only access resources

[spring raptor]

It will be dedicated entities, so no aliasing

[dire aurora]

Getting ComponentId would be a huge improvement at least

[spring raptor]

Will add, worth having.

[spring raptor]

Under the hood of the possible Resources will be UnsafeWorldCell, just like inside EntityMut.

I will play with it, will see what will be more ergonomic.
The use case is clear.

[dire aurora]

Oh also, these are definitely not a priority, but I had some ideas for the event channel issue and those send rules

For event channels I think we should just make the transport responsible for which channels get used, that way crates like replicon that don't split things up can merge all events except ReliableOrdered. Replicon can then just add an ID before each event if the channel has more than 1 event

For send rules there are some implications for rollback (and things like interpolation), we basically need the rules to be deterministic and have it be possible to evaulate on the client. That way it's easy to only check for ticks that could've been sent in the first place, rather than making wrong assumptions based on gaps in data 🤔

[spring raptor]

I like this idea a lot!

[spring raptor]

Could you elaborate on this one?

[dire aurora]

So basically lets say we replicate a component with the policy of "Send once", a crate like my rollback crate needs the ability to check "Would this tick even have a valid authoritative value?". Same thing for something like "Only send once per second"

Basically if an entity can be confirmed, without that value matching the server, it needs to be possible to check for that

Otherwise we'll see issues like "Well every previous value is confirmed so Position(0, 0, 0) must still be valid, when really the intended way to network it is: Send the starting position, use deterministic replication afterwards

[spring raptor]

Ah, makes total sense

[echo lion]

@spring raptor to get complete packet loss info about update messages you’d also need to include the total update message count in init messages

[spring raptor]

And I need to record lost events.

But I think that packed loss is better to measure on transport level

So we better to provide an API for transport to notify us about it.

[dire aurora]

We don't even necessarily need full info, as long as you can tell it's happening and ideally also have some trace logging for it (partial world updates are also what can cause some of the most confusing things too)

[spring raptor]

@dire aurora Do you also adjust your input window based on packet statistic?

[spring raptor]

Opened issues or commented about everything we discussed to not forget:
https://github.com/projectharmonia/bevy_replicon/issues/338
https://github.com/projectharmonia/bevy_replicon/issues/337
https://github.com/projectharmonia/bevy_replicon/issues/336
https://github.com/projectharmonia/bevy_replicon/issues/331#issuecomment-2422903660

[dire aurora]

I don't adjust the input window but I do adjust the timing of prediction, tho I don't currently check for packet loss in any way

[spring raptor]

We probably should provide information in backend-independent way from RepliconServer/Client...

[dire aurora]

I don't even use the transport for this info. I just collect stats on when inputs arrived relative to the server's tick, and send a processed version of said stats back to the client in an event once per second

[spring raptor]

That's an option.
But transports like renet already does that. If we add an API for backends to provide this info, this could reduce some calculations on your side.

[spring raptor]

More like a set of features that are necessary for other crates, especially for rollback.

[dire aurora]

And fleshing out some planned features with some extra details needed to not undermine things like rollback and interpolation

[dire aurora]

I got type erased rollback loading working, and it's scary

[spring raptor]

With the new component history? Great!

[dire aurora]

I have this test and it passes ... Still need to write some code to clean up the prediction history after rolling back, make it play nicely with change detection (currently it calls insert_by_ids for every non-removed component value), add tests for some of the other cases, add the version that inserts new values, make storing and loading run in parallel across entities, and write tests for things like dropping values

After that it'll finally be in a state where I can test it in my game if I don't do any of the annoying edgecases (like fast fowarding a ton of frames, like if the client gets paused for whatever reason)

Te test above also does everything in only one archetype move:

ArchetypeId(1)
- rollback::Predicted
- rollback::history::predicted::PredictedHistory
```to```
ArchetypeId(2)
- rollback::history::load::tests::A
- rollback::history::load::tests::B
- rollback::Predicted
- rollback::history::predicted::PredictedHistory

[spring raptor]

Awesome!

[dire aurora]

insert_by_ids kind of stopped feeling scary now that all my code deals with type erased stuff anyway 😂

The diff so far
9 files changed, 2601 insertions(+), 1 deletion(-)

[spring raptor]

That's huge!
Is it docs and tests or the code?

[dire aurora]

This new code has 47 tests, and most of them aren't very short ... The rest of the rollback lib (which still includes the old history stuff) has 46 tests

I can probably cut down on some of the lines by adding and sharing a few more test helpers

[spring raptor]

That's a lot of tests

Cool!

[dire aurora]

Also a bit of duplication in a few lines and some clutter from that TypeId dance I have to do in write functions

[spring raptor]

Sometimes duplication in tests is fine. Depends on kind of duplication, but when tests written with a lot of abstractions, they are hard to read.

[dire aurora]

Most of the duplication I have is app setup and preparing or asserting some data in a somewhat readable way compared to what I get fully writing it out

I also have 182 unsafes in the new history vs 4 outside of it 😅

A lot of those are in tests to call unsafe APIs however

[spring raptor]

Cursed 😅

[dire aurora]

Yea, it really is cursed. The data structure is unusual, large portions of the API are unsafe, and it's a lot less straightforward than the old solution

But centralizing all this logic to a few systems that can easily manage all the storing, loading, cleanup, etc really opens up some better performance potential

And ofc no more double archetype moves every time you insert a predicted component

Or triple even if you also get an authoritative history

[spring raptor]

Yeah, I think it's worth it.

[spring raptor]

@dire aurora , I finally released the input crate (#crates message).
I will port my game to it, and after that, I'll start working on the features we discussed for Replicon.

I think it would be great to have an integration for your input queue crate in the future 🤔
Here’s how I imagine it. bevy_enhanced_input has a concept of conditions. You provide an API to mark actions and specific conditions as networked. Then just listen for action events and feed them into the input queue.
On the server, before applying a state, evaluate the conditions marked as networked. I decoupled state from mappings, you just pass a new state and it will trigger all necessary events.
I know you're busy with rollback right now and using LWIM for your game, but check it out some time 🙂

[dire aurora]

The input queue just works on whatever components you give it, provided they implement InputTrait which specifies some stuff about repeating inputs when missing and such. The bigger challenge would be networking it efficiently, sending a Vec<bool> wouldn't be very efficient compared to making a staticly defined bitmask containing any values and condition bits you want

And I guess networking stuff securely, you can't replicate any condition that isn't simply input from the player, otherwise you can do silly things like tell the server you're on the ground when you're not

[spring raptor]

Sure, conditions don't need to be replicated. I imagine that you just register them and server knows about them.
You mentioned that stuff like "checking if player is on the ground" better do on both client and server.
My input crate provides an abstraction to define conditions like this. I didn't do it specially for networking, that the way my input crate works. I just think that it could be reused.

[spring raptor]

Yeah, I remember that it doesn't depend on any input crates. That's nice.
I just think that having an integration crate (like bevy_replicon have bevy_replicon_renet) would be neat.

I like having things as separated crates maintained by different people, this reduces the bus factor and usually have positive impact on quality since authors rely on each other work.
But having easy integration between them is important.

[dire aurora]

Yea and there's also the part where not everyone might want to use an integration. Sending only inputs that remain valid after the checks works so much better for prediction that allowing the server to have those inputs succeed despite them failing on the client

[spring raptor]

there's also the part where not everyone might want to use an integration
Is it because of the mentioned bitmasks for efficiency? Do you think it's impossible to provide an efficient abstraction?

Sending only inputs that remain valid after the checks works so much better for prediction
Yes. I believe that conditions are beyond the scope of your crate. I would expect them to be either in the input crate and reused in the integration or implemented entirely on the integration side (depending on the input crate).

[dire aurora]

Do you think it's impossible to provide an efficient abstraction?
Either needs a fairly clunky API or some sort of compromise in terms of performance. Maybe a bit better if for serialization you at least have bitcode instead of bincode with default settings I guess 🤔

[spring raptor]

Ah, I see 🤔

[crisp stump]

@spring raptor where is your input crate sorry? I have been trying to use leafwing but it misses some important clicks and such

[spring raptor]

Released it just yesterday 😅
#crates message

[crisp stump]

@spring raptor You're a god. I have a HUGE project. What's something it might be explicitly better at while I give it a go?

[spring raptor]

See this message: #1297361733886677036 message 🙂

[crisp stump]

Ta

I'm just tackling this now in my blueprints system so this is awesome timing

@spring raptor could you please speak quickly to these 1/2 issues:

https://github.com/Leafwing-Studios/leafwing-input-manager/issues/527

https://github.com/Leafwing-Studios/leafwing-input-manager/issues/626

I never saw it, now I'm getting it alot, probably have my systems badly-ordered but yeah

Haven't exactly had the time to debug it all, have resorted to standard bevy for now which is sad because also had to rewrite half of LIM

[spring raptor]

With my crate you react on actions in observers, you don't check state. So shouldn't be a problem.

[crisp stump]

Perfect

Last question: have you considered clashes?

Not hugely important to me but have had to fight them a few times now

[spring raptor]

Yes, actions executed in order and there is also BlockedBy condition.

[crisp stump]

Sick

[spring raptor]

While integrating the input crate into my game, I realized that having world access inside conditions and modifiers is unnecessary.
It's more ergonomic and faster to access world from observers.
I originally modeled the crate after Unreal Engine, where world access is available everywhere, but in Bevy, it doesn't make as much sense.
Actually using the crate sometimes gives a better perspective 😅 That aside, I'm quite satisfied with the API!

So, conditions will become purely input-based, and if there is an integration with the input queue in the future, gameplay-based conditions should be handled on the integration or input queue side.

After finishing the migration, I will start working on the features for Replicon.
I'm planning to finish it soon, but note that I'm pretty bad at estimating time 🙂

[shrewd sundial]

Hi, I'm upgrading from v0.24 to the current. It looks like bevy_replicon::client::client_mapper::ServerEntityMap no longer exists? What should I replace it with?

[spring raptor]

Hi! It's here, you probably disabled the default features. You need to enable client feature in order to access it in recent versions.

[shrewd sundial]

I'm confused. It seems like client/client_mapper.rs was removed upstream

https://github.com/projectharmonia/bevy_replicon/blob/964ada3a1f9889b26a19fb43dcdb8da19a85b241/src/client/client_mapper.rs
vs
https://github.com/projectharmonia/bevy_replicon/blob/v0.28.4/src/client/client_mapper.rs

Is it all done by bevy_replicon::core::server_entity_map::ServerEntityMap now?

[spring raptor]

ClientMapper is a helper to map entities for components that implement MapEntities trait.
It's no longer needed since mappings now done by contexts passed inside ser/de functions:
https://github.com/projectharmonia/bevy_replicon/blob/master/src/core/ctx.rs

What are you trying to do?

[shrewd sundial]

I'm updating an project from Bevy 0.13 to Bevy 0.14 and upgrading all the other crates to. bevy_replicon 0.24 was being used in some example code to sync positions of physics objects from server to clients with .replicate_with::<Transform>()

[spring raptor]

You don't need to manually map entities, replicon does it for you automatically.

[shrewd sundial]

Cool so I just have to serialise <Transforms> manually in the new version?

Ah I see, I must of serialised the transform because scale wasn't needed. Looks like replicate_with documentation has a similar example to. Thanks for the help and simultaneously sorry for wasting your time

[spring raptor]

Yes. Alternatively you can serialize Position and Rotation from avian.

No problem, you are welcome.

[shrewd sundial]

I guess whether to serialise or not would depend on how concerned one is with bandwidth and whats happening when replicate()/replicate_group() is used instead. Do they poll continuously or only send a event when there's a change? When there is a change does it send sizeof all components, sizeof the changing component or the changing component member?

[dire aurora]

Assuming you have a standard serialize impl from the serde derive, any time change detection is triggered, that component is sent until the client acknowledges a tick that was newer than the latest change

If you use a custom serialize function you can cut down on how big the serialized format is, but the behavior should otherwise remain the same

[spring raptor]

You can also add client diagnostics plugin (https://docs.rs/bevy_replicon/latest/bevy_replicon/client/diagnostics/struct.ClientDiagnosticsPlugin.html) to see how much data you send/receive.
You will be able to read it from Bevy diagnostics subsystem or directly from ClientStats resource.

[sullen lance]

is this the right place to ask about bevy_renet2? i didn't see a separate thread, apologies in advance if this is wrong.

i noticed that even when i'm not sending any packets, renetcode2 is constantly logging Received packet from server: Payload, and bytes_received_per_sec is about ~90 kB/s. am i missing something obvious?

this is using memory channels, so maybe there's some weirdness there. i'll try setting up a regular socket, but curious if anyone know in advance. thanks!

[echo lion]

I can't recall/find why this happens, but if you reduce the tick rate then it should go down. renet has the same issue with high tick rate. There is a https://discord.com/channels/691052431525675048/1038137656107864084 channel if you want to discuss there.

[dire aurora]

@spring raptor Does replicon have any solution yet for the "networking related entities" problem? In my case: I want my player and all status effect entities to arrive at the same time so no mispredictions are created from loading in partial authoritative data

[spring raptor]

Not right now. Spawns and insertion will be grouped, but updates are not.
Sounds like a valid use case, I will open an issue.

Still busy with the migration to the input crate, but hope to finish it soon.

[spring raptor]

@echo lion I just noticed that renet2 is missing from the assets list on https://bevyengine.org/assets/#networking
You may want to submit a PR to https://github.com/bevyengine/bevy-assets

[spring raptor]

Finished the migration to my input crate, switching to replicon now.

@dire aurora What features are the most important for your game and the rollback crate? I will prioritize them.

[dire aurora]

For my rollback crate the changes to write and getting confirms for entities that didn't get updates. For my game the related entities thing

Oh and if you want to tackle that insert_by_ids thing, this is how I do it. Though there are some weird edgecases with this where if changes happen while the buffer is being filled (which is always the case in my rollback code for example, since loading can use a custom function that gets Commands) you need to put those in a separate buffer and apply them after the insert batch ... Probably not an issue for replicon since inserts and removes should be happening only for init messages

There's definitely room for improvement there tho, it's too easy to just .deref_mut::<T> in the write function when you should only .deref_mut::<MaybeUninit<T>>

[spring raptor]

Will take a look!

[spring raptor]

@dire aurora started with the simplest: https://github.com/projectharmonia/bevy_replicon/pull/343

[dire aurora]

Code looks good to me, maybe missing a test or added assertion that the correct ID is passed, tho the data was already around so maybe that already existed somewhere

[spring raptor]

Yeah, I just passed what I used internally.

Now let's see what we can do with the world or resource access... 🤔

[spring raptor]

Regardless the world access in write function, I see the following options:

1. Pass World and Entity. Very flexible, but it requires an entity lookup for each comonent (comparing to getting the EntityMut once and reusing it for each component).
2. Introduce special WorldResources that allows to get resources only. Should solve @dire aurora problem, but may require rework when resource as entities will be introduced. I assume that they will provide a way to check if an entity is a resource, so we should be able to adjust 🤔
3. Pass EntityWorldMut instead of EntityMut. It can provide world access via world() method.

I currently inclined to do 3.

[dire aurora]

3 sounds the most reasonable, but are the structural changes EntityWorldMut allows a potential issue? 🤔

We could of course do some custom wrapper that's just EntityMut + &World so you can't make structural changes of course

[spring raptor]

I think you could do even structural changes here.
But I think it might be a bit weird to insert resources or despawn entities in a writing function.

[spring raptor]

So maybe this actually better 🤔

For your use case you need only read access to the world?

[dire aurora]

Yea, tho inserting components is a usecase I had with my old approach so I wouldn't be surprised if others using write functions would have that too ... But then realistically we want to offer an API that can use batching in some way anyway

[spring raptor]

Right!
So while having full write access won't cause issues, having something like DefferedEntityMut just a nicer API.

I will do non-batching PR first (with the old commands approach) and consider batching in a follow-up.

[spring raptor]

The PR is up: https://github.com/projectharmonia/bevy_replicon/pull/345
If you have a better idea how to name the new entity - you are welcome :)

[spring raptor]

Hm... I think we can allow DeferredWorld here 🤔 Will it be more convenient?

[dire aurora]

Could be more useful maybe yea, it has a few features on top. And would also feel more consistent with that name

Is EntityMut + World actually allowed tho? What if you get the entity again? 🤔

[spring raptor]

I think so, world() borrows the entity, so you won't be able to get an EntityRef from the world, mutate EntityMut and then use the ref again.

If I understand correctly, EntityMut doesn't provide world access because it's used in queries.

[spring raptor]

Hmmm, with DeferredWorld users can access commands, but they actually needs to use the commands from context 🤔 Could be confusing.

I think I will keep &World for now and back to it when I replace commands from context with batching.

[spring raptor]

About related entities.
Right now we iterate over the world and serialize data sequentially as we go. We do it on archetype basis.
To support related entities I see two options:

1. When entities are related, define non-splittable parts. Even if it's bigger then 1200 bytes, we won't split it. The downside that between related entities could be other entities, we won't be able to control it.
2. Use separate buffer for each entity. And concatenate them before sending in efficient way. Slower, but arrays could be buffer then 1200 bytes only if actually needed by related entities. Not sure how much is slower... We need to copy our data into Bytes anyway. Instead of a single memcpy, it will be entities count times more.

This relevant only for update messages.
BTW, do you actually hit the limit of 1200 bytes when you send updates for your abilities?

Thinking about it now because in order to include the number of updates efficiently for history, I need to implement https://github.com/projectharmonia/bevy_replicon/issues/186.
And if I planning to implement it now, it's better to see if a bigger rework for messages is needed.

[dire aurora]

I can't hit 1200 with just a single entity if that's what you're asking. I do easily hit the 1200 per frame of course

One approach I've been experimenting with in my rollback crate is building the archetype cache in such a way that fetching them is efficient (just a simple sparse array, one array with the archetype as an index, which points to an index in the list of archetypes), then using a separate list for what to archetypes to actually iterate over. If we have a component on the entity that is being grouped we can build the cache for them without iterating over them at the top level, those components could then have hooks that add the ID to a component on the "parent" which makes them add those entities while iterating over it

We could then build one single buffer like right now, and only let it decide splitting after a whole set of entities, and the entity updates can otherwise still run as they do now (skipping it if nothing changed, etc)

[dire aurora]

In my case the main reason I do it that way is to track removals, if the archetype id is not the previous archetype id, that means there are components that might've been removed and those histories need a single Removed entry, after that they never get touched written to again unless they move again into an archetype where they do exist. I am still missing the cleanup code (in fact my whole rollback crate is in a much less "crateable" state right now with all the subtle missing features from the new histories)

[spring raptor]

Good suggestion!
Great, then I don't have to rework buffers. Will proceed with 186, then with the number of updates and after that I will take a look at related entities.

[spring raptor]

There is one more advantage of using multiple buffers.
Instead of using u16 for sizes, we can write them as varints. In most cases users don't replicate more then 127 entities, it will cost one byte less.
It will require multiple memcpy, but it's fairly cheap 🤔

Maybe it's even faster. Right now I need to copy each serialized entity into each buffer anyway due to different visibility settings 🤔

Will try and benchmark it

[dire aurora]

With per-component visibility you'd also need to serialize it per-entity (or at least per combination of visibility groups)

Serializing and copying should be relatively cheap however, most of the cost in the approach replicon uses would be in creating buffers and iterating over things that don't need to be sent

[raw idol]

aeronet_replicon v0.9.0 (#crates message) has support for bevy_replicon v0.28.4 (haven't started migrating to v0.29 yet)

[raw idol]

should I bother updating to 0.29.0-rc.3 or wait for 0.29.0?

[spring raptor]

Only if you planning to support Bevy RC. There are no breaking changes from the replicon side.

@raw idol Could you open a PR to add your crate to the list of messaging backends?
https://github.com/projectharmonia/bevy_replicon?tab=readme-ov-file#messaging-backends

[raw idol]

opened: https://github.com/projectharmonia/bevy_replicon/pull/348

[spring raptor]

@raw idol also highly recommend to open a PR to https://github.com/bevyengine/bevy-assets
To display it in https://bevyengine.org/assets/

[raw idol]

will do, probably tomorrow though

I'm hoping to catch any bugs, particularly with the replicon integration first, then release an 0.10 and put it up on bevy assets

if anyone is using websockets or webtransport with replicon right now, and could switch their backend to aeronet_replicon and see if there's any issues, I'd be grateful

[spring raptor]

It's fine to put stuff early, more users - better testing.

[raw idol]

that makes sense to me

[spring raptor]

Went with a hybrid approach: a single continuous buffer where all clients store ranges to that memory (reusing serialized bytes where possible), and the resulting message with an exact preallocated size is constructed from it.
This approach finally allows me to use varints for everything and makes the implementation more straightforward. As part of this, I also addressed https://github.com/projectharmonia/bevy_replicon/issues/186.

All of this makes it a bit slower than my previous approach (a separate buffer for each client with entity and component serialization reuse), but the reduced message size is totally worth it.

Almost done, need to merge small refactoring first and write internal documentation!

[echo lion]

Might take me a bit to get this reviewed

[spring raptor]

The PR is up:
https://github.com/projectharmonia/bevy_replicon/pull/352

Sure, no rush! Take your time.
I would recommend to start reading from update_message and init_message modules, they contain a lot of internal documentation.

After that, I plan to do a small follow-up with one more optimization (see the PR description) and then provide the history API requested by NiseVoid.

[echo lion]

You said it has efficiency improvements, what do the benchmarks say?

[spring raptor]

It optimzies for the message size.
The performance is a bit slower, but we getting a free byte per changed entity and several bytes per message.
For example, in my stats test we using only 17 bytes for a test message instead of 31.

[echo lion]

Ah ok, sounds significant

[spring raptor]

Added it to the PR description.

[thorn forum]

hello ! when a troop get killed in my game, i wanted to despawn it on both server and client side for it to be faster in case of latency. but since the entity has been removed replicon doesn't find it when the server replicate the despawning, and panic.
is there a way of doing this ? or the best way is just for the server to despawn it ?
thanks

[echo lion]

Hmm I don't think we need it to panic on despawn failure. I'll open an issue.

https://github.com/projectharmonia/bevy_replicon/issues/355

[spring raptor]

Hi! I would recommend to send a despawn event to server and wait for replication back.
Because if on server you decide to not despawn an entity (you need to validate what client sends), you will end up in desync.

And yes, we shouldn't panic because of it.

[thorn forum]

thanks for the answer! it's because I wanted to run the damage logic in both client and server for it to be faster to update

and the same for the despawn event

but it's okay I'll just do everything server side

[spring raptor]

I would suggest to avoid predicting damage. Otherwise players will be annoying to see visuals and then realize no damage were done.

Deaths are even worse. Imagine making a headshot and then seeing the enemy wake up because of a misprediction 😅

For example, in shooters you usually predict movement and skill/weapon visuals.

[thorn forum]

thanks for all your answer. though for my game I think it's fine since it's a 2d grid based game, like clash of clan something like that, so no risk of being a desync except if packets are lost

[spring raptor]

You still can get a desync if you predict things.

For a TD game I wouldn't predict anything. Users simply won't notice the latency.
Games like Starctaft or Warcraft use lockstep and no one notice. There are plenty of TD maps for them.

[thorn forum]

alright fair enough, thanks for everything and have a good day !

[spring raptor]

Working on the follow-up and it changes more things then I expected.
@echo lion have you started reviewing #352? If not, I will probably push the changes directly into #352.

[echo lion]

Have not started. You can push them in, it's fine

[spring raptor]

Great, I think it will be easier to read them as a single change.

[dire aurora]

The best one has to be predicted victory screens tho. Seeing "You won" or "You lose" for a fraction of a second before the game realizes that was a misprediction isn't very fun 🥲

[thorn forum]

so games really do that lol ?

[dire aurora]

It's a somewhat common issue in fighting games when they first get rollback networking. In the case of P2P rollback netcode you kind of have to predict even victory, but showing it immediately with no delay is what causes such issues there

Predicting death can be fine depending on the game, but despawning wouldn't be the way to go in case they do turn out to be alive

[spring raptor]

Pushed the changes! I think it's ready for review now. As always, no rush.
Originally, I planned to make a small change to store the number of updates that NiseVoid needs, but ended up completely reworking the serialization 😅
I really like the changes - not only the code is nicer, but the messages are now much better packed.

Tomorrow, I'll replace the reported panic with a warning (or maybe an error) and start working on the update history.

[echo lion]

Ok, not sure when I will get to it. Implementing something for a bonus slide for the bevy meetup thursday, not sure when I will be done with it.

[brittle mulch]

@spring raptor Wait but it isnt ideal to predict bullets?

[spring raptor]

Are you referring to storing the number of updates? This just allows rollback crates to assume that an entity's value hasn’t changed for a specific tick if all updates for that tick have been received.
Needed for better prediction.

[spring raptor]

@dire aurora we currently don't send anything if there are no changes. Is this fine for you or do you need us to send an empty message to know that there are no updates?

One more thing. If there is any insertion or removal for an entity, we move all updates for this entity into our reliable message. So you will need to check ConfirmHistory on the entity first and if it's not confirmed, look into the global history.

[dire aurora]

Hmmm, ideally it would still send something so the other side can know nothing changed rather than just assuming the data is late or the other side stopped sending for whatever reason, but it's laos a bit of a waste of bandwidth 🤔

[spring raptor]

I can provide a plugin option if needed, so users who don't use prediction won't be affected.

[dire aurora]

Yea that seems reasonable. Another strategy would be relying on batching. It's rare for replication and events to both fully stop so if there are events replication could still be sent as no changed and be in the same packet

Tho ofc with renet that won't actually happen but whatever

[spring raptor]

I will go with an option then since most people use renet and I'm not sure what other backends do under the hood

[spring raptor]

Better idea. I can enable this automatically if any marker registered with need_history.

[dire aurora]

In my case I don't think that would work since the server has no clue the rollback crate exists 🤔

[spring raptor]

Ah, okay, then plugin option it is!

[spring raptor]

@dire aurora I have an array with this struct:

#[derive(Clone, Copy, Debug, Default)]
pub struct TickMutations {
    all: usize,
    received: usize,
}

Would you prefer to have access to it directly or just get u64 as a mask with ticks that fully received?
Or do you need contains_any and contains?
Or both? :)

[dire aurora]

Not sure what the struct and the fields do, but in most context I do need a mask at least

[spring raptor]

It tracks the number of received and all messages. If received == all, then all mutate messages for this tick were received.
I think you don't need this implementation detail, just a mask with contains and contains_any like with ConfirmHistory?

[dire aurora]

Yea, mask and maybe events for the mask changes should be all anyone ever really needs I think

Well, maybe for stats someone might want it, but that only makes sense if replicon doesn't track stats itself

[spring raptor]

Hm... Not very convenient. This option needs to be enabled on both server and client (since it changes the message).
So it needs to be an option for RepliconCorePlugin. And users will need to remember to enable it.
Not very ergonomic.

Maybe I could make it configurable via resource. This way you will be able to set it automatically.
What I don't like about it is that it will be a resource with a boolean option...

Another approach is to make it a cfg feature, but it's a pain to maintain for me 😅

[spring raptor]

Never mind, it's possible to read plugin data from other plugins via get_added_plugins.

[spring raptor]

No, other plugins can't change the data of other plugins.
Will make it a resource then. I liked the setting on plugin because it implies that this option shouldn't be changed at runtime.

[echo lion]

If the resource is read-only, you can have a pub(crate) method to update it via an App extension. Then only plugins can modify it.

[spring raptor]

Thanks, did as suggested and I like how it ended up!

@dire aurora The PR is up:
https://github.com/projectharmonia/bevy_replicon/pull/359
Opened as a draft since I want you to confirm that this API fits your crate first. But all tests pass, the change should be fully working.

It's based on a branch that is based on another branch, but it's because koe were busy with the Bevy meetup. But it won't affect you, you already try the branch.

[spring raptor]

Also opened a PR that provides a way for backends to supply statistics:
https://github.com/projectharmonia/bevy_replicon/pull/360

[dire aurora]

Hmmm, no events for the global confirms? Also wondering if there's potential problems that could be caused by how it runs in the middle of everything, rather than at the end (by storing all confirms to a map of Entity -> previous mask, inserting only if it's missing, then sending those events with the previous mask at the end)

I can't think of a way that would go wrong with rollback but if there was say an interpolation crate that did something other than updating a target tick in those observers things could get weird 🤔

[spring raptor]

MutateTickConfirmed event is global, it's triggered when all mutate messages received for a tick. HistoryConfirmed event is for an entity.
Or maybe I should named it GlobalHistoryConfirmed? 🤔
I used MutateTickConfirmed name for event and ServerMutateTicks for the struct since it track ticks for mutate messages 🤔

Ah, I emit MutateTickConfirmed for each received message. Will adjust the code this evening. Forgot to check the triggered events in tests, it's a bit of a pain 😅

[spring raptor]

Agree. I will collect entities and ticks into a Vec first.

[spring raptor]

Pushed the changes, should be good now.

[echo lion]

I'll start reviewing today

[spring raptor]

@dire aurora sorry for the ping, I just wanted to make sure that you saw the message. If you are busy with something else, it's fine, take your time.

[dire aurora]

Ah, I somehow did read it but not see the question nor the "pushed changes" ... I'll see if I can test it in a bit

I think the name is fine, I just didn't see it while looking at the code because it was nicely hidden between a big impl block and unit tests 😂

[dire aurora]

Had to first do some migration work for import paths and using the ComponentId that is now provided ... Diff ended up quite large because of the latter, but there is a decent reduction in code

12 files changed, 141 insertions(+), 182 deletions(-)
```Will test the stuff from the new PR in a bit, need to eat something first 🙃

[dire aurora]

Manage to implement all of it in code, but when I start the server I get:

bevy_replicon-6598e87287870c1c/bc1b7b9/src/server/replication_messages/mutate_message.rs:197:13:
assertion `left == right` failed
  left: 1159
 right: 1150

I also think with the current way they work HistoryConfirmed and MutateTickConfirmed would make more sense as buffered events

Like I just have two duplicate observers doing something that would be orders of magnitude faster if batched 😅

fn rollback_for_X_confirm(
    trigger: Trigger<X>,
    mut rollback_target: ResMut<RollbackTarget>,
) {
    let event_tick = trigger.event().tick.get();
    **rollback_target = rollback_target
        .map(|tick| tick.get().min(event_tick))
        .or(Some(event_tick))
        .map(RepliconTick::new)
}

[echo lion]

Is this in #359? #352 doesn't appear to have a bug in that spot.

[dire aurora]

Yea, only happens when enabling the new feature

[spring raptor]

Interesting...
When I create a message from ranges, I calculate its size in order to preallocate the memory for Vec. And I put debug_assert to make sure that it matches the resulting size to avoid reallocs.
Looks like the message is bigger then I expect (1159 bytes instead of 1150). Probably I miss-calculate this when the feature is enabled.
I wondering why I didn't catch it in my tests 🤔 I guess I need another test for a big message. I already have a test like this, but when the feature is disabled. I will investigate and add a test.

[spring raptor]

Agree, I will rework.

[echo lion]

@spring raptor just this left https://github.com/projectharmonia/bevy_replicon/pull/352#discussion_r1859329698

[spring raptor]

Yep, answered!

Finally merged the messages rework 🎉
@echo lion you are a GOAT! Such thorough reviews even for +1,716 −1,327 ❤️

Need to merge a tiny follow-up and I'll start addressing the history PR for NiseVoid since it depends on it.

[spring raptor]

@dire aurora fixed, added more tests and made events buffered: https://github.com/projectharmonia/bevy_replicon/pull/359

[raw idol]

has there been any discussion about a ChannelKind::UnreliableSequenced? I think this would be really useful for inputs

[spring raptor]

Renet had it, but removed at some point.

Probably it's because you can just sort the messages on your own on receive.

[raw idol]

can renet give me some ordering info for the received message, like a sequence number?

[spring raptor]

No, you have to add it yourself.

[raw idol]

and replicon doesn't have this info either?

doesn't it have an ack system?

[echo lion]

Right now acks are an internal implementation detail with no guarantee of stability.

[spring raptor]

We use them only for sending mutations.
For other channels acks handled by renet.

It would be great to have an API from transport for acks, but no transport exposes it or allows optionally include it for unreliable messages. So we have our acks at home for one small thing.

[raw idol]

got it, I just implemented this channel kind in aeronet because I already have a seq u16 used for identifying messages, so I use that for UnreliableSequenced

[raw idol]

aeronet_transport does :)

although, all packets and messages are ack-eliciting, which is not great. I still need to do some work on the protocol

[spring raptor]

If you need it in Replicon, PR this channel kind. Godot has it as well.
For backends that don't support it we can panic.

[spring raptor]

That would be cool to utilize.
But I would like to wait for more adoption for such API for other crates. Otherwise we just break other people backends since they don't expose that.

[raw idol]

I'll probably just leave it for now, and reimplement it manually.

I'm not sure you can design a generic acks API which works for all backends, it's a very transport-specific thing. Might be possible though.

[spring raptor]

I mean if your backend supports UnreliableSequenced, we can just add one more enum variant to let you send messages using this kind of channel.

[raw idol]

idk how I feel about it panicking on other backends though

[spring raptor]

Yeah, not easy thing to implement. I don't have any idea what API to suggest.

[spring raptor]

That's up to backends anyway 😅
But I would panic then silently fail in this specific case.

[dire aurora]

If the panic can be at startup, getting a panic is no problem at all

[raw idol]

I investigated making a generic acks API. The main problem I faced is: what type do you use for storing a "message key" / "message seq"?

• could be a u16, but not all backends use u16s / they might need more data
• make it a bigger, more generic u32/u64, but this still might not be enough for some backends, and you lose type safety
• make it a Box<dyn MessageKey>, but now you have an indirection + you lose type safety + you can't clone/copy the key
  in the end I decided it wasn't worth it

[raw idol]

Yeah, it should be doable on startup, since that's when channels are setup anyway I think?

[spring raptor]

I think Joy suggested something about acks API for backends.
But yeah, I also not sure how it can be done nicely.

[raw idol]

if I'm feeling selfish, I'd say just make it a u32, since that's how big my MessageKey type is:

pub struct MessageKey {
    /// Lane index on which the message was sent.
    pub lane: LaneIndex, // u16
    /// Message sequence number.
    pub seq: MessageSeq, // u16
}

But, I think a u64 would be enough for any transport. And if that transport's message keys are too big for a u32 or u64, then it can instead use the key as a key into a map, rather than just as a POD

[dire aurora]

Had to jump trough some hoops to test this, but I created a situation where the client always mispredicts, while the server sends no updates, previously this would have caused either no rollback to happen at all, or the repredictions to be wrong because it didn't snap back to old server state. Just to be sure it only assumes that when it's valid I set some very unstable latency and high packet loss

Previously all the spheres that are now jittering would've fallen trough the floors (it doesn't have a collider on the client), and the moving ones would suddenly reappear

And with no latency or packet loss it looks almost as if the ground has collisions, except you can't do anything with a grounded check

[spring raptor]

Great!
It's an extreme situation 😅

So I can mark it as ready for review for koe?

[dire aurora]

Yep

[spring raptor]

Would you like me to draft a new release after that?
Or you need 0.15 anyway?

[dire aurora]

Don't really need a release since I have it working now, and migrating to 0.15 is indeed the next thing I'll need to do on my game's repo (I'm working on some SDF tree stuff in the meantime)

[echo lion]

I will review the history PR once I release bevy_cobweb_ui for 0.15 (hopefully I'll do that later today).

[spring raptor]

Okay! It's a small one, it's mostly just tests.

After it I planning 2 more: the suggested "change" - "update" rename and visibility check for despawn sending.

Maybe 3 - remove depreceted stuff.

[spring raptor]

PRs are up!
After them I will push the last one for the message rename, already have it in stash, just need to merge others first. And we are ready for the 0.15 update.

[spring raptor]

@dire aurora merged a few renames that occurred between releases. Just letting you know since it affects you.
HistoryConfirmed -> EntityReplicated
MutateTickConfirmed -> MutateTickReceived
ServerMutateTick -> ServerUpdateTick

Going to draft a new release right now.
Just merged Replicon's 300th PR! 💪

Are you currently busy with SDF and the migration to 0.15?
Just wondering how the rollback and input queue crates are doing.

No rush, I just writing the release announcement.

[dire aurora]

Rollback is now in a slightly better state, but I need to migrate it to 0.15 and get rid of the awful single file prediction logic to test the API well. InputQueue I haven't touched in a while but I know it has some weird problems, especially surrounding configuration that is actually unusable and not loading inputs received from the server correctly (which is important for rollback, but should probably be configurable for inputs that should work delay-based)

[spring raptor]

Can't even imagine how hard is to test something like this 😅

[dire aurora]

Testing rollback is almost entirely vibe-based. You add some latency and packet loss and just see if it plays as intended. For input it's easier, since it's possible to verify by hand that the inputs are being networked and loaded correctly (but currently part of that code lives outside of the crate which is rather impractical)

[spring raptor]

Makes sense.
You mean that for input queue you have some test helpers?

[dire aurora]

More that it is just trivial to log the values and messages and know that it is working correctly (and that also means you can be reasonably confident if tests pass, tho of course system ordering could still be wrong in practice)

[raw idol]

/// Sets the round-time trip for the connection.
///
/// <div class="warning">
///
/// Should only be called from the messaging backend.
///
/// </div>
pub fn set_rtt(&mut self, rtt: f64) {
    self.rtt = rtt;
}

this should specify what unit it's using in the docs

I assume seconds

[spring raptor]

I agree. I "stole" it from renet, which also doesn't specify 🤔

https://docs.rs/renet/0.0.16/renet/struct.RenetServer.html#method.rtt

[spring raptor]

But yes

[spring raptor]

Opened PR to adjust the docs

[brittle mulch]

Might i ask yo ua few questions?

Why did you estabilish replicon in this sort of mutliple crates style, why not make it monolithical

[spring raptor]

There are several reasons:

• Modularity: Not all games need prediction. For example, some require Rocket League-style rollback, while others might use "classic" per-entity rollback. This approach allows people to choose what to opt into.
• Code reuse: There are already several good messaging libraries. Instead of writing messaging from scratch, I believe it's better to provide an API and reuse existing people's work.
• Maintenance: Maintaining a single monolithic crate would be a nightmare. This way, more people can contribute by writing their own abstractions.
• Bus factor: If something happens to the maintainer, it’s easier to fork a small crate or replace it entirely. Fortunately, in our case, we have two maintainers with full admin access 😅

In short, we’re doing similar to what Bevy does: encourage community to create third-party crates for their needs. Of course, our ecosystem doesn't have that level of adoption, but we're slowly getting there.

[thorn forum]

hello ! i would like to have a snappy client experience: for example when i place a building, im currently sending a event to the server, it spawns it, and it's replicated to the client. i would like to also spawn it directly on the client, and then the server replication comes. is this possible with replicon ?
i guess i would the replication to map to existing entities instead of creating a new one when replicating to the client ?
thanks !

[dire aurora]

You can use https://docs.rs/bevy_replicon/latest/bevy_replicon/server/client_entity_map/struct.ClientEntityMap.html for this

[thorn forum]

ohh it was a dumb question 😁 i got confused and thought it was for replicate_mapped and EntityMap something.
thanks !

[spring raptor]

Not dumb at all! I see how it could be confusing.

[covert fog]

what is the current recommend way to replicate a scene that's loaded from an asset?

if you replicate something like a handle to the scene, then the spawned scene's children wont be replicated.
but I have a hard time thinking of a different way without building the scene up in code instead of an asset

[spring raptor]

I would replicate a path to the scene from the handle were loaded.
Another way is to create a hash asset path and use it as identifiers to determine which scene to load. But it's more complicated.

[covert fog]

Okay, but how do i get replicon to replicate the spawned scene entities. if the client and server spawn them seperately, they have to be mapped after the fact

[spring raptor]

If you want to spawn them independently, then you need to notify the server about your entities by using https://docs.rs/bevy_replicon/latest/bevy_replicon/server/client_entity_map/struct.ClientEntityMap.html

[dire aurora]

This likely wouldn't actually work as it's more likely the server tells the client "Here is an entity with a scene and here are all the replicated things below it", and then the client needs to load the scene and somehow wire those up together 🤔

[spring raptor]

Ah, right 🤔
Maybe split scene into sub-scenes and replicate them. Not very efficient, tho.

[thorn forum]

hey ! does

            .replicate_mapped::<Projectile>()
            .replicate_group::<(Projectile, Transform)>()
``` supported ? im spawning a `Projectile(Entity, Entity)` (the source and target) on the server, but on the client the entities are wrong (doesn't exist, invalid), i tried logging the `Projectile` and on both the server the client the Entities inside of it are the same, which i dont think is normal.
 i have implemented MapEntities:
```rust
impl MapEntities for Projectile {
    fn map_entities<M: EntityMapper>(&mut self, entity_mapper: &mut M) {
        self.0 = entity_mapper.map_entity(self.0);
        self.1 = entity_mapper.map_entity(self.1);
    }
}

i may be missing something obvious, as its the first time im mapping something.
its working normally in singleplayer, but not server client.
if there's isn't something obvious i missed, ill try to explain in more detail :)
thanks

[spring raptor]

You enabled replication with mapping for Projectile and additionally enabled non-mapped replication when transform is present.

You will need to create a replication group manually in your case with replicated Projectile and Transform.

Something that could be nicely solved in the future by supporting With. I have an issue open for it.

[thorn forum]

thanks for the answer ! by manual replication group you just mean .replicate them one by one ? i have now a .replicate_mapped for the Projectile, and a .replicate for Transform and it mostly work now

[spring raptor]

No, no, just create a struct and implement ReplicationGroup trait manually for it.

You can also make this struct a bundle, but it's optional.

[thorn forum]

ohh ok thanks I'll look into it

[spring raptor]

That's if you want to replicate Transform only if Projectile is present.

Otherwise yes, you could just call replicate_mapped for Projectile and just replicate on Transform.

[thorn forum]

ReplicationGroup is a little too complicated for now, and replicate Transform works and without drawback for me as it need to have a Replicated component anyway to be send

[covert fog]

So, am i correct in seeing this as a pretty big unsolved problem?

(Difficult to get into this topic without sounding like and ass, i appreciate all your work on this project)

Basically, im wondering if replicating scenes assets is something you think we should eventually have a recommended way of doing, or if theres a differnt way of going about this entirely

[spring raptor]

Am I understand correctly that you have a scene made with Blender (a level, for example) and want to replicate it including entities?

[covert fog]

Yes

Lets say, some of the entities have physics turned on, for example

[spring raptor]

@covert fog Makes sense.
Bevy just doesn't have an official scene workflow right now, which is why users usually don't bring it up 😅

I think we should support it. When Bevy has an editor, I expect the workflow to be similar to what you do in Blender.

Right now we provide very low-level primitives: map entities received from server and tell server which entities correspond to which from client.
We need a higher level abstraction on top of it for scenes. By any chance you interested on working on it? Could be in a separate crate for faster prototyping.

[covert fog]

I could take a crack at it. My project is blocked networking anyway.

[spring raptor]

It would be great!
Let me know if you have any questions about the Replicon's low-level API.

[dire aurora]

This one might actually be somewhat difficult. If we want the server's entities to map to the client's entities without having the server send basically the entire scene to the client, we'd need the server to send a sort of map from some identifier within the scene to the server entity

With the obvious issue here being that afaik scenes don't have any such identifiers 🤔

We could make some replicon-specific concept for it, that would also allow the server to know what needs to get Replicated without the client immediately adding that (which would make it hard to debug when the mapping goes wrong)

[spring raptor]

The scene have its own entities that later mapped into spawned entities. Maybe we could use them as IDs? Just send the client their mappings to server entities, which client wire to its own local entities after spawning the scene?

[dire aurora]

Hmmm, could work, but we'd also have to consider what would work when we get bsn, and I'm not sure if it has this particular strange feature

These features are also generally hard to get right, because bevy's current scene system has very few users and even fewer that would want to replicate some data on entities in the scene but have both sides load the scenes directly otherwise

I do use something similar to bevy's scene system in my game but it's just to instantiate entities based on some ID that points to a scene, which is probably a far more usual approach when you can't easily hand author a game scene

One thing that could maybe work is using Names relative to the scene root

Other than that I'm not seeing anything in the bsn proposal that could help us find entities unless we define our own component to identify them and mark them for replication

There might be some plans to tag them so they can be found again for features like reactivity tho, I'll ask in the workgroup

[spring raptor]

If BSN doesn't have entity identifiers, like in the current scene format, it should be possible to manually assign index-based identifiers and use them 🤔

[dire aurora]

Not if all we get is an already spawned scene with no way to tell what it came from, otherwise we'd need a replicon-specific spawning mechanism and that would really suck

Also feel like a better solution should be possible that wouldn't entirely break apart once you start introducing hot reloading

[spring raptor]

Agree, we need to cooperate with the scene workgroup on it.

[echo lion]

I expect the only solution is:

1. Client must register all scene entities with replicon on spawn. Need parent:child mapping + node entity:root entity mapping.
2. Server must annotate non-root scene entities with the scene root node entity. Then on client, cache them (with parent/child relationship from server) until enough client spawns have been registered to reconstruct the scene path from root to node (i.e. to trace out the hierarchy in order to map things right).

This design would actually work pretty seamlessly on cobweb ui I think, if I made some small changes to scene spawning. Since scene spawning relies on a trait + allows 'node initialization' baked into the scene spawner, so all I need is two replicon-specific spawners for server/client. And hot reloading is non-destructive.

[echo lion]

Actually you could probably use a deterministic child-node-id approach to avoid hierarchy reconstruction. Like increment a counter while spawning the scene, and assign that to a special replicon component on the entity (including root entity).

Like ReplicatedScene. Then on client use an Added<ReplicatedScene> system to collect mappings.

Would work for static scenes but idk about dynamic scenes that change structure (spawn/despawn children).

[echo lion]

You could use a marker component ReplicatedSceneRoot to annotate the root node, then in a system traverse the scene hierarchy to add ReplicatedScene components to children. Works for both server and client, no need for special scene spawning.

So replicon would need to add a section to UpdateMessages for scene mappings. Like [root node : [scene nodes]] and indices are inferred from the array of scene nodes.

I guess for destructive hot-reloading you could overwrite mappings in replicon client.

[spring raptor]

That could work!

But I think @dire aurora wanted to somehow make it without special handling from replicon side? Not sure how that would work.

[echo lion]

Although actually on destructive hot-reload you need to completely re-replicate entities... so I guess you'd need a server event that tells the server to reset scene entities. And on the client, some way to map the old scene root (which is despawned on hot reload) to the new scene root.

[echo lion]

Or maybe instead of a mapping section, use an init section like [root node entity : [scene node data]] to avoid duplicating the scene node entity.

Is the motivation for replicating scenes like spawning a mutable environment map?

To make that useful wouldn't you need two representations of the scene, one for server and one for client? Seems hard to maintain.

[dire aurora]

The less manual intervention to get things working the better, but of course there is only so much we can do before requiring users to put in some effort

[spring raptor]

I just starting to doubt that it's possible withour an explicit senction in the message...

[spring raptor]

I think the idea is to spawn a level that already made in Blender or future editor and then replicate entities from it.

[dire aurora]

Yea I'm fairly sure we'd need an explicit part in the spawning message, just like with the client spawning thing

You just don't want it to work the way the client spawning does, since the server most often spawns things before the players (like for examples players might join later and then still spawn these things (which then also potentially happens AFTER receiving the data))

[echo lion]

Does this make sense for a headless server?

[dire aurora]

Most likely you would have a scene with scenes nested in it, and swap out only the lowest levels for client vs server to avoid spawning useless crap or thing that don't even exist in the server's set of bevy features

[spring raptor]

Depends on the game. But I think server could decide which level to load and then tell the client about it

[echo lion]

Is there any prior art for scene replication in unity/unreal/godot?

[dire aurora]

Godot essentially always replicates scenes, and it avoids this problem by doing it by path

Which is obviously not an ideal solution either, but we could have it as an option and allow people to specify names for changing entities in replicated scenes

[spring raptor]

It somehow just works in Unreal 🤔

Right now I'm busy with my game, so I would expect people who interested in this feature to write a proposal.
And research prior art.

@covert fog so it will require some changes on replicon side, I don't think it can be implemented as a third-party crate. But if you are interested - write a proposal. Scenes are not well-explored thing in Bevy.

[dire aurora]

Cart said there is no current implementation of any tracking mechanism for bevy, so at the very least I doubt we can anticipate what a feature like this would look like. So the only bevy-native solution without custom replicon features would be using Names relative to some scene root, but I'd imagine we might need the replicon-specific component anyway because we don't want to replicate every entity from a scene this way, but rather only changing entities in scenes that are spawned on both sides

[covert fog]

I think attaching Replicate to entities in the scene (either in the asset or pragmatically) for the children you want replicated makes sense

my understanding is that currently, there is no reason for the client to attach Replicate to an entity (except for replicon doing so internally when things are replicated)

Maybe I'll look at how lightyear does prespawning

[dire aurora]

Yea, it would be weird for the client to do so, since afaik it's just added on the client if it was replicated by the server. But the server would still need to know the entities need to be replicated, and that it should be synchronized based on the scene instance rather than as a standalone entity

[covert fog]

Well the scene entities would be replicated from the server as well.

So we'd end up with duplicates on the client.

If Name is replicated we can match them up by path and fix the mapping

I guess the issue then is recreating the heirarchy on the client side. My understanding is that parent and children components are not replicated by default?

(Though id think replicating them when both parent and child are replicated should be possible)

Hmm parentsync exists, I suppose we need that on every entity in the scene for this to work

Alternatively we could have a ReplicateId component that gets added deterministically by a system/asset

And just use that for matching

[spring raptor]

No, it needs to replicate parent-child relation.
When you spawn the same scene on client (for example, level), the hierarchy is already correct.

For scenes we just need to map replicated server entities into client entities correctly. Once you establish the mapping - it should just work.
But we need a mechanism for it. If you write a proposal, I can help you implementing it. I will be able to guide you through the code and tell what changes will be needed or implement it myself. But I need for someone to do a research first. Take a look at prior art.

[covert fog]

No, it needs to replicate parent-child relation.
I'm not sure what you mean by this. "No" as opposed to what?

1. "replicate parent-child relation" is what I thought ParentSync did
2. The rest of what you said implies we don't need to replicate the parent-child relation (assuming we have a way to do the mapping that doesn't use the hierarchy info)

[spring raptor]

I suppose we need that on every entity in the scene for this to work
To this sentence ^

[spring raptor]

Yes, this is correct

ParentSync will be needed only if you planning to make changes to the scene hierarchy after instantiation.

[covert fog]

Looks like what light-year does is generates a hash based on tick and an entities components, which is used for mapping. I'm going to look into the implementation more. So far my proposal is going to be, roughly, to have a ReplicationId component to be used for mapping, and a provided (optional) system for setting this to the hash of the hierarchy path (with the entity id of the root scene in there somewhere).

I'll write it up in a git issue when I'm further along.

[spring raptor]

Lightyear don't have this feature.

[covert fog]

It has prespawning

which is similar

[spring raptor]

We have the exact same thing

Both crates support entity mappings and you can sync the hierarchy. Also you can pre-spawn an entity on the client.

But it's unrelated to scene spawning syncrhonization.

[spring raptor]

Here: https://docs.rs/bevy_replicon/latest/bevy_replicon/server/client_entity_map/struct.ClientEntityMap.html

From what I understand, you want to spawn a scene on client and on server and then send updates to its entities.
It's different. You need to re-wire entities from client-spawned scene to entities that replicated from scene from server.

[covert fog]

Well, yes, the client might spawn the entities after or before the server (race condition)

[spring raptor]

That's already supported.
But if you want to spawn a scene and start synchronizing children's components - we need a mechanism for it.

[dire aurora]

Especially with lightyear's approach it really wouldn't work well for this since you might spawn a scene on both sides, but the server would inevitably have to start before the client and the hash for spawning would be guaranteed to be off. Similarly prespawning them on the client then sending IDs to synchronize them as replicon does is an extra round trip that really shouldn't be necessary

The biggest case the consider here is actually: The server already has a running scene, it sends the entity for the scene and all children, these children don't have anything to synchronize to yet, but when the scene is instantiated they would ideally be applied immediately rather than spawn at the default values and then jump to some new values after a round trip

Would be ideal if we could inject existing entities into the scene and have the scene be spawned as an insert_if_missing but afaik there are no features like that, so I'd imagine we're going to need some buffering mechanism for cases like this

[spring raptor]

Yep, I had the exact same thoughts, outlined them in this message: https://github.com/projectharmonia/bevy_replicon/issues/376#issuecomment-2535507860

[dire aurora]

Relying on the Entity from a scene might be problematic since afaik BSN won't have that feature

[spring raptor]

I would expect BSN to have some identifiers, otherwise it won't be possible to apply entity mappings when you spawn an entity with a component that references an entity inside the scene.

[dire aurora]

I think BSN uses names heavily for purposes like this

We could do that too, but there are some potential issues. We'd either need names to be unique per root, or have more names in the hierarchy above them to identify them

If we go for that approach it would essentially be what Godot does, replicating things by hierarchy paths. I'd imagine Godot just sends hashed paths but it's network impl is pretty bad so maybe it sends the full strings 🙃

[spring raptor]

But you want to put a component that stores Entity that points to an entity inside a scene, what it will reference? 🤔 Will they replace all Entity with strings? But how it will work for custom types?

[dire aurora]

I think you reference it by name but it will resolve it for you itself

As for custom types I'd imagine Construct plays a role in this too

[spring raptor]

What if we generate our own IDs based on index via iteration? Should be cheaper and simpler then hashing.

[dire aurora]

Generating IDs would be hard since we probably don't get to be between all BSN and spawns, and doing it on an already spawned hierarchy would be inconsistent (entities aren't sorted) and unreliable if the hiarchy changes at all

Hashing names shouldn't be very expensive tho, especially if we only hash the path when the path or name changes (which should become very easy to do next release with immutable components)

But we could alternatively have users specify it too, or have a way for users to manually specify it (in case the name constantly changes and isn't a stable identifier)

[spring raptor]

I thought about inserting our identifiers as components right after spawning the scene 🤔

[spring raptor]

I mean that with varint encoding index-based IDs will take less space. I would expect hashes to be u64 or something like this.

[dire aurora]

Oh. Hashes can really be any size as long as it gives sufficient protection against collisions. If the scope is only a single scene the hash probably doesn't need to be u64. We could also sort all the paths under a root and number them, but that would break if we reload the scene or spawn something extra that could be picked up as a new item for the scene

There is also the case of nested scenes that aren't spawned as one scene, which would need to reference only their closest parent rather than the most upper parent, but that shouldn't be very difficult to handle

If we go for the sorting approach we could probably put in some barriers so things don't get picked up by accident

[spring raptor]

I wondering what to do in a case like this:

- scene root
-- NameA
-- NameA

The same applies to mappings inside the scene. If instead of entities they use names, how they determine which entity to use if there are multiple entities with the same name? 🤔

[dire aurora]

Yea that's the big pitfall of this approach. You need something to identify them uniquely, but we don't really get one so we either require names to be unique or we introduce our own thing for a unique identifier

Of course it's unlikely you would name everything in a scene the same, but if BSNs get support for loops it becomes easier to make this mistake

[spring raptor]

But I wondering how Bevy itself solves this problem. If they are not planning to use unique identifiers, how do they map entities inside components if names are not unique?

[dire aurora]

If you reference by name they would have to be unique within that scene I'm pretty sure 🤔

Not exactly hard to give things a name that is at least unique by its full path

[spring raptor]

🤔 Let me ask them what they think about name uniqueness...

[covert fog]

I think duplicate names aren't that unlikely to occur within a scene, but probably refer to identical entities (ex. spawn 10 pigs).

in which case, we can just match them up any which way.

as for an alternative, the question is whether there is a deterministic way to give them an index.

[dire aurora]

For an already spawned scene giving them an index in a deterministic way would be impossible. But we can turn names into an index (possibly as a setting, since it's mostly just a bandwidth optimization at the cost of breaking hot reloading)

[spring raptor]

It looks like there will be a way to uniquely identify entities in a scene.
So either sorting and generating index-based IDs or reusing theirs (depending on what they decide to use) should work well for us.

[flat holly]

Hi is it somehow possible in replicon to disable the ack reply from clients? I am using a custom websocket approach and therefore everything is ordered

[dire aurora]

I don't think there currently is, but I'd imagine a setting to make replicon assume things got acked and just not send it would be possible, might get a bit more complex if we want to support usecases where both websockets and webtransport are enabled for a server however

[spring raptor]

Sounds useful to have a plugin option for this 🤔

[spring raptor]

Maybe we could make it a per-client option to support this.

And default into the plugin one.

[echo lion]

bevy_renet is pretty standard. You can also use bevy_renet2 from the renet2 repo.

[spring raptor]

Yep, it's because Renet hasn't been released yet.

You can try 0.15 branch: https://github.com/projectharmonia/bevy_replicon_renet/pull/17
Or switch to Renet2. It's almost drop-in replacement.

[spring raptor]

Opened an issue to provide a built-in backend: https://github.com/projectharmonia/bevy_replicon/issues/379
Something very simple, just to run examples.

[raw idol]

MPSC backend?

[spring raptor]

Not convenient for examples, requires using the same binary.
IPC will be nicer.

[raw idol]

You're right, I didn't consider that. I would say a raw localhost TCP socket then. Is IPC cross platform?

[spring raptor]

TCP will also work, yeah. But some people may start using it for real networking, unlike IPC 😅

Yes, it should be cross-platform. I think I saw a rust lib for it.

[dire aurora]

Make it UDP, just slap a channel number in front of each packet and done

Then hide it behind a feature named please-give-me-bad-transport-code that only examples and tests would dare to enable

Oh and make the plugin log a warning when it is loaded that tells people that it should not be used for anything except examples and tests

[spring raptor]

I were thinking about a separate crate inside the workspace. Just to make sure that it works exactly as third-party transports.

[dire aurora]

Sure, that's an option too, as long as it's clear that you're pulling in some netcode that doesn't belong in a production game unless for some reason it can only ever run on localhost

[spring raptor]

Yep, I planned to not publish it on crates.io and give self-explanatory name, like bevy_replicon_example_backend.

Unrelated, but it looks like naia is still being actively developed.

I rarely see people mention it 🤔 I suspect this is because the crate is engine-agnostic (supporting both Bevy and Macroquad via adapters), which makes its API less ergonomic. That, combined with the lack of documentation.
I wondering if it's possible to write an integration for bevy_replicon. The Naia socket could become a new backend, and instead of the current bevy adapter (or alongside it?) have an adapter for Replicon.
My game doesn’t need prediction/interpolation, and when I’m not working on it, I spend my time on replicon itself (I have a bunch of requested features planned).

Still, maybe someone else might be interested. It would be great to have a classic shooter-style rollback crate.

[dire aurora]

@spring raptor I'm thinking of pulling my networking crates out to its own crate and making an example, which means I'd need to make a lightweight transport, want me to pick up the work for that example transport?

[spring raptor]

It would be great, would be awesome to have it!

But maybe it will be easier for you to just pick any existing transport? Just less work for you.

[dire aurora]

Looking at the code for the renet one it seems easier to make one from scratch than actually get renet or some other crate to work 😂

Well only if you make a hacky one that doesn't have the features replicon expects of course, which would be fine for an example

[spring raptor]

Ah, I meant not for the example transport, but for your example.
I.e. set something like bevy_replicon_renet(2) = ... in dev-dependencies.

And when we get the example transport, you will be able to switch to it.

But if you want to contribute it now - sure. This way you won't depend on releases of other crates which is a plus.
And I will be able to put examples in the main repo :)

[dire aurora]

Do init updates rely on reliable transport at all?

[spring raptor]

Yes, when I pass them to the transport, I expect them to be always delivered.

This is why I suggested IPC.

Another option would be to use UDP for unreliable and TCP for reliable channels 🤔

Maybe UDP + TCP will be better because this way users will be able to introduce packet loss via standard OS utilities.
But I don't have any strong preference.

[dire aurora]

I'm thinking we can just make a resource with packet loss settings

It's just for examples anyway and the tools are kind of obscure so most people won't be able to test them

Then we can just change what packet loss settings do for "reliable" packets (buffer them longer before giving them to replicon as if a resend happened)

[spring raptor]

Yes, that's also an option. Will be more user-friendly.

[dire aurora]

Oh look, setting up an app with networking and rollback is actually pretty easy

App::new()
    .add_plugins((
        DefaultPlugins,
        RepliconPlugins,
        RollbackPlugin::<Tick>::default(),
    ))
    .init_resource::<Tick>()
    .run();

But wait what is this

#[derive(Resource, Clone, Copy, Default)]
struct Tick(u32);

impl From<RepliconTick> for Tick {
    fn from(value: RepliconTick) -> Self {
        Self(value.get())
    }
}

impl From<Tick> for RepliconTick {
    fn from(value: Tick) -> Self {
        Self::new(value.0)
    }
}

[spring raptor]

I thought you get rid of custom ticks 😅

[dire aurora]

It's now a user-specified tick

Can't really not have a tick source since rolling back requires a reversible clock

[spring raptor]

But can't you edit RepliconTick directly? 🤔

[dire aurora]

Client doesn't have RepliconTick but it's also clunky to use a type you can't extend everywhere

I mean ideally we'd have a well-designed SimulationTick in bevy and no one would need to extend it, but until that happens being able to slap on whatever derive you need and do possible translation steps is a lot simpler for game code

Also RepliconTick might satisfy the trait bounds if anyone wants to use it 🤔

It's Copy + From/Into

[spring raptor]

Maybe default to it with = RepliconTick in generics? Just for simplicity.

[dire aurora]

Wouldn't work great since then it will just confusingly panic on not having RepliconTick

[spring raptor]

Do you need the same resouce for both crates, rollback and the input queue?

But yeah, we ultimatively need a tick for it in Bevy

[dire aurora]

Both take a tick source yea, and ideally you'd set them to the same resource and use some provided systems to actually load inputs for resimulated frames

[spring raptor]

Maybe we could have a resource in replicon for it as a temporary measure?

[dire aurora]

That would just create the same confusion as when people thought RepliconTick was necessary for something on the client

With the user providing the resource you have the nice benefit that people immediately understand that it's their tick resource and there's just a few crates using it for something

[spring raptor]

I thought about creating a separate resource for the client

[dire aurora]

The tick here is on both the client and server, and it might not match replicon's ticks too. Like if your update rate for replicon is 20Hz but your simulation runs at 60Hz you'd need to return different values

[spring raptor]

Then initialize it in core, just as a separate type.

Replicon won't use it, but just to share across third-party crates.

Another option is to store it inside the rollback crate and pass on initialization into input queue crate

[spring raptor]

Because defining it manually sounds like an extra boilerplate. Most users don't really care about what tick is used internally.

Just a suggestion :)

[dire aurora]

Well that's the thing, they kind of have to care, because rolling back anything time-based that doesn't use the tick is going to misbehave

Timers and the like are all pretty problematic with rollback and input queues

[spring raptor]

Ah, I see.
But I would still add a tick for it in Replicon until Bevy doesn't have a proper one 🤔 Just to make it less boilderplate to setup, defining it manually won't solve the misuing time-related stuff in rollback.

[dire aurora]

It does sidestep all the problems with people thinking replicon does things it doesn't do with ticks

Like synchronizing clocks

My crate doesn't currently do that either, if I defined a tick people would probably think I do tho 😂

Will probably pull that out of my game's code and make something configurable for clock sync, but people would still need to add the crate

(The example will need clock sync to work after all)

[spring raptor]

Even if it's not defined by the crate, people might assume all kinds of stuff when you pass it to the plugin... The resource just need to be properly documented.
But I'll leave this decision to you 🙂 Feel free to let me know if you decide to include it in replicon. Just saying that I'm open to it. Yes, it's not a perfect solution, but it's a small win for ease of use.

[candid summit]

new to bevy_replicon, been going through the docs and I have some basic questions about how to approach my use-case

Each user on a separate app is a standalone server with local client by default, keep the logic as simple as possible. Not concerned with cheating. If I want one of these apps to remote connect to another in any way, do I need to remove the (default in my case) server resource and add a client resource to automatically attempt to connect the two users?

[dire aurora]

Could you go into some more detail on what exactly you are doing? Running a standalone server by default sounds rather unusual 🤔

[candid summit]

Sorry, still getting my network vocab down. I mean like a listen server or host server style setup, so single player is just a local authoritative server with a local client. I prefer to avoid splitting the app into different versions and just have the same code work whether client or host.

To be clearer, I still want an authoritative server, no P2P or anything.

The app is cooperative tooling, so cheating isn’t a concern here, just security.

[dire aurora]

Ah okay, a listen server makes more sense. It's not uncommon to only start a server or client once someone actually decides to host or join, and afaik you would do so by adding/removing the client/server transports. I don't think you actually need to touch the replicon server/client (if you do it won't be able to clean up things it might need to clean up anymore)

[spring raptor]

Yes, this will work as you expect.
But you will need to cleanup the state before connecting.

[candid summit]

I’d like to have a server always up, so that any new features can be designed once; to interface with a server of some kind, local or not, and not have to worry about any complications from having it also work in a server less state

[candid summit]

Is this the intended way for a listenserver to connect to another listenserver as a client?

[dire aurora]

Running a server or not shouldn't make any real difference to the hosting app

If there's a server it sends state, if there isn't it ... Well, doesn't

[spring raptor]

You need to remove the server resource of your transport first.

In the quick start guide I tried to explain how to design your logic in a way that works for any mode.

[candid summit]

You have a point. Network stuff is my weakness, so I was trying to remove as much complexity as I could

I see. In my case I’m using the renet2 integration, and it attempts connection when the resources are added. So this isn’t the right thread to ask

[spring raptor]

If you have a server resource - you replicate from this instance. If you have a client resource - you will receive replication.
App logic is not related to it. And it's usually abstracted from it.

[candid summit]

The automatic stuff is what was tripping me up, I think. I was looking for something like run_server(); or connect_to_host(addr); and wondering what I was missing

[spring raptor]

This depends on the backend you using.
But I think most of them work similar to renet.

[candid summit]

I think I get it now. Thank you both for the help!

[dire aurora]

I was working on my networking example with the example transports ... Got them working now but because it doesn't have any fancy connection management I ran into some fun edgecases: I need to send a message to get it to connect, the easiest way is client events, but they are extremely brittle when you try to send events at the same time as connecting. Had to introduce a whole State thing for connection just so I can send an event when connecting 😂

No renet required!

(Yes the client car is still client side, so it doesn't show up on the server)

This is an interestic panic

thread panicked at bevy_replicon-0.29.2/src/server/replication_messages/update_message.rs:262:21:
assertion `left != right` failed
  left: UpdateMessageFlags(MAPPINGS)
 right: UpdateMessageFlags(MAPPINGS)

[spring raptor]

Awesome!

[spring raptor]

Interesting. It tries to send an update message with only mappings.
But it shouldn't be possible 🤔 If you send only mappings without the entity data, it means that the client already received this entity.

[dire aurora]

Ah I think I see what I did wrong

The entity doesn't have Replicated

[spring raptor]

Ah, then I probably should change it from panic to something else

I didn't know that it's possible to trigger 😅

[dire aurora]

Yea probably because it only panics if there are no other entities to be replicated either

I thought I would be able to get everything up and running first and then port clock sync later, but I realized I need clock sync for rollback and the input queue to even work correctly 😅

I'll just build a hack that will break when latency is introduced for now

[spring raptor]

Maybe just unite them in a single crate? 🤔
Or do you think it's possible to use input queue separately from prediction?

[dire aurora]

Definitely would be possible to use input queue without rollback, like if you are making a lockstep game, or just do some very naive prediction that doesn't bother with rollback

Or maybe someone eventually makes shooter-style rollback for replicon and wants to reuse my clock sync and prediction

[spring raptor]

Got it, makes sense

[dire aurora]

Shouldn't be too hard to add another crate to this repo I'm working on tho, there's already 4 crates might as well make it 5 😂

[spring raptor]

The example transport could be submitted to the replicon repo 🙂

[dire aurora]

Yea I'm already not counting that one

[spring raptor]

And with clock, input queue and rollback it makes it 4. Which one is 5th?

[dire aurora]

It's rollback, input_queue, entity disabling (on top of DefaultQueryFilters), and entity management (to help with reusing entities, especially for predicted spawns)

[spring raptor]

Ah, interesting!

it's cool that it's modular.

[dire aurora]

Kinda just had to do it that way so I can have rollback without a bevy fork, and so I can have a single API in shared code without needing entity disabling in my game server

[spring raptor]

Just migrated my game to Bevy 0.15, and without observers, our events API feels incomplete.
I think observers will be especially useful for networking events.
I plan to work on them soon, but first, I want to make some changes to my input crate.

[brittle mulch]

Events die out after a few frames i think such a alteration would increase realibility

[dire aurora]

Yea I also ran into issues with that before, having a more reliable option could be very nice

[spring raptor]

Just opened a PR to migrate some our built-in events to triggers first:
https://github.com/projectharmonia/bevy_replicon/pull/384

Now working on API to create networked trigger-based events.

[thorn forum]

hello ! im having problem with ParentSync :)

use bevy::{prelude::*, state::app::StatesPlugin};
use bevy_replicon::{
    prelude::{ParentSync, Replicated},
    RepliconPlugins,
};
fn main() {
    let mut app = App::new();
    app.add_plugins((MinimalPlugins, StatesPlugin, RepliconPlugins))
        .add_systems(
            Startup,
            (
                (|mut commands: Commands| {
                    commands.spawn(Replicated).with_child((
                        ChildMarker,
                        Replicated,
                        ParentSync::default(),
                    ));
                }),
                (|child: Single<Option<&Parent>, With<ChildMarker>>| {
                    // the child has a parent here
                    println!("child's parent right after spawn: {:?}", *child);
                }),
            )
                .chain(),
        )
        .add_systems(
            Update,
            |child: Single<Option<&Parent>, With<ChildMarker>>| {
                // the child doesn't have a parent anymore, if ParentSync::default() is on the child.
                println!("child's parent in Update: {:?}", *child);
            },
        );
    app.update();
}

#[derive(Component)]
struct ChildMarker;

gives:

child's parent right after spawn: Some(Parent(0v1#4294967296))
child's parent in Update: None

the child's parent has been removed ! it only occur with ParentSync, and when spawning early like Startup, or StateTransition (how i stumbled upon this issue)
thanks !

[spring raptor]

Thanks for the report!

Looks like it happens because we store changes into ParentSync in PostUpdate: https://github.com/projectharmonia/bevy_replicon/blob/96cd34f5558d7446f5429d76175b5dfd1c9249ef/src/parent_sync.rs#L40
But apply the changes in PreUpdate: https://github.com/projectharmonia/bevy_replicon/blob/96cd34f5558d7446f5429d76175b5dfd1c9249ef/src/parent_sync.rs#L34
So when you spawn an entity in Startup, it applies changes first.

Maybe we can fix it by utilizing hooks 🤔 It's late night for me, will be able to take a look tomorrow.
If you have some time now, feel free to take a look. It's a very simple logic located in a single module.

[thorn forum]

thanks for directing me. its also late, and ill probably dont have much time in the following days, but ill be happy to implement this fix when i can !
ill try with a observer to update the parentsync when its spawning with the initial parent. i guess it should work. have a good night :p

[pulsar garnet]

Is there any high level stuff written about how replicon works? I'm going through the plugins but there's a lot of different resources, systems etc all working together.

Even stuff like what is ReplicatedArchetypes, is that a performance optimisation, or required to function at all?

Otherwise though really impressed and reading through the source code now 🙂

I have a bevy_ecs only project that I'd like to do network replication on, so I'm looking at replicon to get inspiration to make something with just what I need from it.

[dire aurora]

ReplicatedArchetypes is mostly a performance thing. There are two naive approaches to replicating an ECS world:

• Serializing everything that changed every frame, checking what there is to replicate
• Register systems to replicate registered components

The archetype cache thing is something I originally came up with with bevy_bundlication iirc, but the implementation in bevy_replicon is a lot more advanced and polished by now. Basically instead of the above approaches you build a cache for every archetype that needs to be replicated so you immediately know what to replicate. Then you iterate trough replicated archetypes, the entities in those archetypes, and can easily loop trough only things you actually need to check

In bevy_bundlication (which used the "register systems" approach previous) it was about as fast as having only 1 system to replicate things, but multiple systems scaled worse than O(n) while the archetype-based approach scales better than O(n), with say 10 components the performance difference was about 9x faster

bevy_replicon previously did the "serialize everything, checking what to replicate" approach, and the performance gains it saw were much bigger

[dire aurora]

I wonder if it's possible to just port bevy_replicon to requiring just bevy_ecs 🤔

There are of course a lot of features that would require bevy_app or feature flags that you'd ideally enable on the bevy crate itself

[pulsar garnet]

Yea, seems like most of the core logic only requires bevy_ecs, it's just the convenience of plugins and existing stages like PostUpdate to know when to run systems

[spring raptor]

Unfortunately, there is no high level explanation.
But I tried documenting even private stuff to make it easier for newcomers to understand the code.
Feel free to ask questions.

[spring raptor]

Fixed: https://github.com/projectharmonia/bevy_replicon/pull/387
I planning to draft a new release soon, want to include networked observers.

[spring raptor]

Replaced with an error message:
https://github.com/projectharmonia/bevy_replicon/pull/386

[thorn forum]

wow thanks a lot ! it works for my use case. thought it doesn't work if the parent is set before the parentsync component is added, might want to add a warning (see PR)

[spring raptor]

Thanks, pushed a fix with a test case 🙂

[spring raptor]

System builders is such a nice feature!

Just refactored our events logic with it: https://github.com/projectharmonia/bevy_replicon/pull/395
To avoid creating several systems for each event type, we store the IDs of all event resources and process them at once. This is similar to what Bevy does.

However, since we need to access resources by their IDs, we used exclusive systems with a lot of resource_scope.

But with system builders we can define the required access and get rid of exclusive systems!

[pulsar garnet]

ooh interesting, is there good docs on system builder? feels like something I need to look into for that kind of "meta" functionality

[spring raptor]

Yes, it's pretty well documented:
https://docs.rs/bevy/latest/bevy/prelude/trait.SystemParamBuilder.html
There you can also find references for all types that you can pass into it with additional docs. Here is for resources that I used:
https://docs.rs/bevy/latest/bevy/prelude/struct.FilteredResourcesMut.html

[spring raptor]

I see that entity disabling was merged, nice!

[dire aurora]

Just DQF, no entity disabling on top yet, but that should be easier to review/merge now, since it doesn't touch ECS internals

[spring raptor]

I still working on networked triggers.
Originally I tried making them as a separate thing, but it resulted in a lot of duplication.
I also tried extending current events logic to just support triggers, but the logic becomes quite hard to understand.
Right now I settled on just expressing triggers through existing events, like a small abstraction on top. Triggering remote events just fires regular events with entities. That later gets serialized and dispatched on receive.

[spring raptor]

Almost done, working on docs

@dire aurora do you have your example transport ready? I wondering if we can merge it with this release to provide examples in the repo 🤔

[dire aurora]

I have this, wouldn't really call it finished but it does work 🤔

[spring raptor]

But how reliable channel will work for UDP? In replicon we assume that reliable channel is actually reliable 🤔

[dire aurora]

It doesn't, but since it's for examples and not to showcase the fact that transports have reliability that shouldn't matter much

[spring raptor]

If it for examples anyway, maybe better to use TCP instead? It would be a bit more expected to have always reliable channels instead of always unreliable.

The PR for networked triggers is up. @echo lion as always, no rush 🙂 But most of the added lines are tests and docs. I don't think it will be hard to review.
https://github.com/projectharmonia/bevy_replicon/pull/398

[dire aurora]

I guess we could just open both a UDP and TCP socket 🤔

[spring raptor]

This would also work. But not sure if we need actual unreliability for examples...
With TCP we just have proper connections, it will look more like a full transport. And replicon won't work weird even on shitty connections, just slow.

[dire aurora]

It will look weird

The HoL blocking and buffering TCP likes doing will bunch up changes in odd ways

[spring raptor]

I think buffering can be disabled, but maybe you are right 🤔

[spring raptor]

What else do you like to add, connection conditioning?

[dire aurora]

Yea, ideally examples can get some controls to add packet loss and ping ... Not a super high priority to get something that just works tho 🤔

Doing it with the example transport also allows us to showcase things like different clients running at different levels of compromised networking, which is usually impossible with standard tools

[spring raptor]

Agree, maybe worth adding it like this and extend in the future?
Also do you want to submit a PR or would you like me to finish the upstreaming?

[dire aurora]

I don't currently have the time to make a PR for it, so it's fine if you upstream it

[spring raptor]

No problem, I will do it tomorrow.
Thanks a lot for working on it 🙂

[spring raptor]

Code coverage something works so weird 😅

It's usually fine, but when you mess with type erasure, it gets super confused.

[dire aurora]

This might be due to not running it with all features diabled that might cause the binary to have a different flow than the code 🤔

[spring raptor]

I run it with all features enabled 🤔

[dire aurora]

I mean features as in things like optimization, symbol mangling, stripping of debug symbols, etc ... There are some you can disable, and it has a big impact on certain patterns of code that it really likes to optimize or inline even when optimization is disabled

[spring raptor]

Ah, that could be the case!
I wonder why cargo tarpaulin doesn't disable this by default.

[dire aurora]

Not sure, but iirc I've had the same issue when I tried it for rollback, I just don't remember how I fixed it since I only used it until my history code had 100 coverage in the previous version (the one without type erasure) 😅

[spring raptor]

In your code you detect connection from a client when you receive a message from it.
Do you just send some welcome message in your examples?

[dire aurora]

Yes, I send a message so that triggers ... It's kind of hacky tho 🤔

Could probably just change the connection code to just send some magic packet the other side will ignore too

Tho ofc UDP is unreliable so there is no guarantee that packet would arrive

[spring raptor]

Yep, that's what I was thinking... How about to send an empty packet?

[spring raptor]

Sure. But probably fine to assume that it arrives since it's just for example.

[dire aurora]

Yea that's possible, there might be some if statements that disconnect on empty packets tho

Would be easy enough to fix that ofc

[spring raptor]

You mean to disconnect on empty packet if the client is already connected?

[dire aurora]

Yea, I had code for that at some point, but I might've changed that

I'm pretty sure the client still does that however

[spring raptor]

Yep!

[spring raptor]

But when we disconnect, we just remove the socket. Maybe worth just error on empty packet in all cases, expect connection 🤔

[dire aurora]

Yea that would be fine, I don't think empty packets should normally happen

Also how do you like the horrible client IDs? 😂

[spring raptor]

It's quite clever, actually 🙂

[thorn forum]

hello ! i have a simple 2d game like clash royale. my game server using renet2 native socket sends 10mb of egress for a 5min game. Since hosting platform count egress, I would like to reduce this amount.
most of the egress are probably transform replicate i guess. its acceptable for my game to not replicate the transform component.
but i would like to initially replicate the transform to position it initially, but not replicate afterward. is there a clever way of doing that ?
or just replicate a custom component like InitialPosition(Vec2) and on the client on spawn set the transform with it ? i would like a better way like replicate only once, or only when spawning it.
thanks !

[dire aurora]

Not anything built in. If you can make do with an initial position and then calculate where it would've gone from there that would work ... But in practice we'd probably want something like https://github.com/projectharmonia/bevy_replicon/issues/331 so we can just set replicon up to only send a position until one is acked, or only send the position once every second

If we get delta compression features we might also be able to hack something like this in 🤔

[thorn forum]

Alright, ill do something like that. is there other things i can do to reduce traffic ? i guess replicon already only send what changed ?

is there a way to "debug" what replicon is sending ? to see if i can optimzie other behavior

thanks !

[dire aurora]

Replicon only sends unacked changes yea. I think the biggest impact on networking would be sending data more efficiently. For example instead of sending Transform as a whole it might be better to have a custom serialization function that sends it as a Vec2 + an angle for rotation. Quantizing data is also an option, but this has some potentially problematic impacts on determinism if it is doing in serialization functions (since the server would have non-quantized values, while the client gets quantized values)

[spring raptor]

Yes, just run with RUST_LOG=bevy_replicon=trace cargo run.

[spring raptor]

#331 is on my todo, but for now you need the mentioned InitialPosition workaround.

[spring raptor]

Ah, you probably want to know which components and entities replicon sends. Right now we trace only the amount of data. I think it worth adding 🤔

[dire aurora]

Yea the only real thing you're gonna be finding out from the debug logging is the packet sizes, and how many separate packets are coming in and going out (https://github.com/projectharmonia/bevy_replicon/issues/336 could reduce that, but changes on the renet/renet2 side could also reduce the impact of using separate channels)

[spring raptor]

@dire aurora without proper connection support when you close an app in example, it won't detect a disconnect 🤔
Maybe it worth to spaw the code to TCP and just disable buffering using this method https://doc.rust-lang.org/std/net/struct.TcpStream.html#method.set_nodelay ? HOL blocking will be present, but for examples it shouldn't be a big problem.
With RPC it will also work fine? Disconnect will be just a file deletion.

[spring raptor]

Remote triggers now merged 🎉
Once we figure out the example backend, I will draft a new release.

[spring raptor]

@echo lion I would also appretiate your opinion on this ^

[echo lion]

I guess my question is: what value does the example provide? It's not like we can recommend using TCP or whatever for actual game code.

[spring raptor]

I wanted an example backend for the following reasons:

1. Run examples from replicon repo

2. Allow third-party crates to run their examples from their repo without depending on a backend.

3. Have a backend example

Testing my new phone with physical keyboard, looks like newline is Shift+Enter 🤔

But you asking a good question. Does it actually provide us any value?

1. People can look for examples to backend crates. It's a bit unusual, but it provides complete example with specific backend they want.
2. Third-party crates could use something like renet2 which updated immediately. And users will have complete example with proper backend.
3. renet2 or renet already are great examples of how to write a new backend.
   So I'm not sure...

[dire aurora]

I guess TCP would be fine too if we can actually disable the buffering ... Might also have to change some things to account for the weird way in which TCP can batch messages (even with nodelay if they are fired right after eachother)

[dire aurora]

Third-party crates can't use renet/renet2/whatever safely, that's why we even wanted to make an example backend. Crates on top of replicon wouldn't be able to migrate until whichever transport they use migrates, and we've seen already that people will wait however long that takes instead of switching transports for the example, which in theory could mean one transport getting abandoned could take half of the ecosystem on top with it 🤔

There is also of course the example thing for replicon itself being wildly confusing and hard to find, but I think that realistically won't be fully solved with just a transport, since we still can't have things like prediction in the replicon repo

But then I guess not every game needs prediction either

[spring raptor]

Maybe it's fine since we don't have packet loss when run locally? Or how would you suggest to mitigate it?

[dire aurora]

When it groups packets it just places message right next to eachother, so we'd probably just have to put the length of a packet in front of it, then read the described number of bytes (or error if that number is unreasonable)

[spring raptor]

renet2 is maintained by one of the replicon's maintainers, so maybe it's fine? @echo lion updates renet2 immediately.
But even with possible bus factor, switching backends should be easy, unless you implement some custom transport on top of it, of course.

[spring raptor]

Ah, I see!

[spring raptor]

Yes, I see how this can be confusing 🤔

[spring raptor]

But even if users try example with our example transport, they will have to adjust their code to their transport. I just think if we pick one, it's less chance that users might need to migrate (because he might want renet2, for example).

[dire aurora]

It's not even really about how easy/hard it is, but purely about how people won't bother. Lets say koe takes 2 weeks to update renet2 because they are busy (which is totally fine), people will just wait for 2 weeks instead of switching, but there is no actual way of knowing ahead of time if it will be 2 days or 2 weeks or 2 months or 2 years

[dire aurora]

I think ideally we abstract the transport away enough for examples to not need to mess with them ... Having people blindly copy what examples do also isn't good for the ecosystem, since that's how we got into this weird situation where everyone uses renet in the first place 😅

I think ideally people make a concious decision for which transport they pick as well as things like prediction (like for example my rocket league style rollback is going to be the wrong pick for a shooter)

[spring raptor]

Agree. I think main advantage (IMHO) is examples inside the repo. I'm pretty sure it confuses newcomers.

But yes, we also need something with proper connection support. Will you swap TCP in your code or want me to finish it?

[dire aurora]

I think the changes should be pretty minor for that so it's probably easier for you to add it ... Mostly just swapping out UdpSocket to a TcpStream/Listener (I think, I've never done TCP in Rust) and going for a "read until no more data is available" approach instead of getting packets

TCP often feels really clunky because of that whole "it's a stream of data" behavior 🥲

[spring raptor]

Okay, I will!
Also never worked with TCP in Rust, only in C++. But I will figure it out.

[dire aurora]

Afaik TCP should function about the same in every language, since it's probably largely the way it is because of how the OS handles it

[spring raptor]

Sure, the difference only in the API design.

[echo lion]

Shouldn't UDP be fine if it's localhost?

Oh it's for detecting connections

[dire aurora]

Yea both are fine if localhost as long as you don't expect the network conditioning to be applied externally (which wouldn't make for a good example anyway, since only like 3% of users are going to have those) ... UDP is just easier until you get to disconnects, where UDP requires tracking connections and timing them out which isn't really worth writing when TCP does that for you

[spring raptor]

Ported my game to remote triggers to try the API myself. Quite satisfied with the result - only made a small adjustment 🙂
Remote triggers also more efficient when the event contains entities, thanks to the handcrafted serialization. I should probably include this information in the docs.

Now, I’ll work on the TCP example backend.

[spring raptor]

TCP is not very convenient to work with. For example, set_nonblocking not applies to TcpStream::connect. I can spawn a thread, of course...
But I want your opinion on 2 alternative options:

1. Use https://github.com/rust-lang/socket2. Unlike standard library, It provides non-blocking connect.
2. Write an integration with a normal backend. Something very minimalistic, no custom transports. For example. this one: https://github.com/benny-n/netcode.

@echo lion @dire aurora ^

I starting to incline towards 2 🤔

[echo lion]

Or you could just use renet and copy-paste the bevy_renet crate (which is quite small).

[spring raptor]

That's also an option. But I think that the netcode crate is more minimalistic. And this way we will have one more backend option 🤔

[spring raptor]

The bevy_renet also requires bevy feature to be enabled on renet. So it won't be direct vendoring, I will have to wrap all types.

If we go the actual backend route instead of the example backend, It won't be considered as first-party. I will advertise it as a minimalistic choice.

[dire aurora]

I don't think connect should really matter for examples 🤔

Even for production games plenty of them just freeze when they connect to a server

[spring raptor]

Yeah, and on localhost it will connect almost instantly.

But writing a wrapper around the mentioned netcode library is just as easy 🤔 And it will be way more useful.

[spring raptor]

The mentioned library is too minimalistic 😅
It doesn't even have channels. Doesn't matter for examples, but I better utilize TCP then.

I also quickly looked for other libraries, but couldn't find anything similar. So TCP is a way to go.

[spring raptor]

Decided to improve the backend API a little bit working on it:
https://github.com/projectharmonia/bevy_replicon/pull/401

[spring raptor]

With resources as entities the API for backends will be so much nicer, can't wait :)

[dire aurora]

Resources as entities would also make rollback and replication for resources so much easier :')

[spring raptor]

BTW, how is the rollback development going?

[dire aurora]

Haven't touched it this week

[spring raptor]

It's totally fine, take a break, maybe you will get some motivation later, don't push yourself 🙂

[dire aurora]

Motivation is definitely somewhat of a challenge because making the example is kind of uh ... boring? ... But the main thing is that I just have a whole bunch of other things competing for time (work, SDF rework, raymarching, SDF collisions, game stuff, etc) so sometimes I don't touch rollback for a couple of weeks

[spring raptor]

Ah, sure, time is the most important asset :(
Feeling the same lately. So much stuff that I need to do.

The backend PR is up: https://github.com/projectharmonia/bevy_replicon/pull/402

@dire aurora I committed the code you sent me and marked you as a commit co-author 🙂
It was quite a good starting point despite we switched to TCP in the end, thanks!

[dire aurora]

In turn bevy_replicon_renet was a good starting point for the example backend I made 😂

[spring raptor]

Looks like I can't collect coverage from TCP backend, it crashes tarpaulin 😅

[dire aurora]

tapaulin loves being weird but this is on a whole other level 😂

[thorn forum]

hello !
im getting this on web using a not very good connection:

panicked at /home/mirsella/.local/share/cargo/git/checkouts/bevy_replicon-28211fcd94f26925/bf760c1/src/client.rs:584:10:
all entities from mutate message should have confirmed ticks

should i open a issue ? thanks

[echo lion]

What transport are you using? renet2 webtransport/websockets?

[thorn forum]

yep renet2 websocket

doesn't happen on native

[echo lion]

Ok it might be a renet2 problem then. For websockets I assumed packets are ordered-reliable and downgrade all channels to 'unreliable' for efficiency but maybe there's a race condition.

[thorn forum]

oh nevermind im also getting the same message on native !

[echo lion]

Ah then it's a replicon bug

[thorn forum]

it might be because i set replicon tick rate to only 20 ?

[echo lion]

Should not matter

[thorn forum]

alright then. thanks for your answers and help ! ill see what shatur thinks and ill open a issue if necessary

[echo lion]

Just to confirm, when calling RenetClient::new(), are you using socket.is_reliable() for the second parameter?

Also, are you using pre-spawned entities with replicon? @spring raptor looks like apply_entity_mapping needs to call confirm_tick

[thorn forum]

true for websocket, and false for native socket

[thorn forum]

no ! the only thing close I'm using is a mapped entity

[echo lion]

Yeah mapped entities are pre-spawned entities

That's probably the bug, will see what shatur thinks

[thorn forum]

oh ok, i thought pre spawned entities was the client spawning an entity and sending it to the server for lower latency.
in my case it's a server replicated component that old a entity.

thanks for the help !

[echo lion]

Wait I'm confused, you are using mapped entities but not pre-spawning on client? How does that work?

in my case it's a server replicated component that old a entity.
Can you explain this more?

[thorn forum]

I do a .replicated_mapped component. the component holds a entity.
the component is only added on the server

[echo lion]

Ah that one

[thorn forum]

by mapped entity I meant just using replicated_mapped, sorry if that's not the right term

[spring raptor]

I think it's fine... There is no way you can receive a mapping and then receive a mutation for a component. You need to insert it first 🤔

[echo lion]

Ah mappings are separate from spawns? Makes sense

[spring raptor]

@thorn forum the panic means that it should never happen, but it somehow happens 😅
Could you try to provide a bit more details, so I could repro the issue?

The error means that you received a mutation for an entity that never received an insertion 🤔

[thorn forum]

its not possible that the packet for the entity creation was lost or arrived later than the packet with the mutation ?

thanks for the answer ill try to find whats causing this

the game server was hosted in docker on hathora (a game server service) btw

[spring raptor]

No, mutation message won't be applied until creation is received... I explicitly handle this 🤔

[spring raptor]

Does it happen randomly for you?

[thorn forum]

it happens in mid late game (1v1, 2d grid, like Clash Royale, not a big game). all the thing possible in the game already happened multiples times before the panic occur, so i guess its random yes if enough things are going on

[spring raptor]

Damn, I hate bugs like this.

@thorn forum may I ask you to try this branch where I disabled the panic?
It's not a proper fix, I just want to get more information to debug the issue.

https://github.com/projectharmonia/bevy_replicon/pull/405

[thorn forum]

oh thanks a lot ! thats awesome to receive such support :)

[thorn forum]

They happen on both client, at the same time:

2025-02-02T12:57:46.399655Z ERROR bevy_replicon::client: ignoring mutations that arrived earlier that insertion for 1140v14#60129543284
2025-02-02T12:57:46.399670Z ERROR bevy_replicon::client: ignoring mutations that arrived earlier that insertion for 1140v14#60129543284
2025-02-02T12:57:46.433095Z  INFO bevy_ecs::system::commands: Entity 1140v14: ["bevy_replicon::core::replication::Replicated"]
2025-02-02T12:57:46.433112Z  INFO bevy_ecs::system::commands: Entity 1140v14: ["bevy_replicon::core::replication::Replicated"]
2025-02-02T12:57:57.124346Z ERROR bevy_replicon::client: ignoring mutations that arrived earlier that insertion for 1218v2#8589935810
2025-02-02T12:57:57.141200Z  INFO bevy_ecs::system::commands: Entity 1218v2: ["bevy_replicon::core::replication::Replicated"]
[...]

ive cut the rest for discord message length, but it continues to happen multiples times with new entities every 2 to 10 seconds.
it means the entity only has the Replicated component ?
im not dong that in my game, and especially the error happen not when doing anything special, when the error happens, the game is doing the same thing as before, spawning the same troop in loop, moving it, attacking the enemy troop, but its always the same troop, always the same target, etc.

i dont understand why the entity exist if the insertion wasn't received ? (i dont know enough how replicon works)

thanks !

[spring raptor]

Replicated is not replicated actually, replicon automatically inserts it (to avoid wasting traffic).
It can have only Replicated component for 2 reasons:

1. You spawned it like this on server. Replicon sends it to clients and clients also inserts this component.
2. Client receives this entity in a component or as a mapping (if you use pre-spawning on client) and also inserts Replicated.
   But looks like later you receive mutation for this entity without any insertions. And it's very weird.

Am I understand correctly that the game itself works fine?

Do you you use inspector_egui? If yes, could you try to take a look at such entities, what they represent?

[thorn forum]

not sure if everything was correctly working. overall it look fine.
im using a mapped component for projectiles, maybe the projectile was broken but didn't see.
im using a Projectile component with the target entity inside. sometime, the Projectile component is spawned AFTER the target entity was despawned. the Projectile component is replicated_mapped to the client. could it be that ?
the bug appearing multiples times every few seconds could correlate with my projectiles systems.
ill see if i can find the entity with inspector egui

i dont know how replicon works, but maybe it expected the Projectile's target entity to exist and so tried to mutate it ?

[spring raptor]

Hm... Let me double check this part:

im using a Projectile component with the target entity inside. sometime, the Projectile component is spawned AFTER the target entity was despawned. the Projectile component is replicated_mapped to the client.

[thorn forum]

there might be more than just the invalid mapped entity, because i think it doesn't happen everytime the Projectile Target entity is invalid

[spring raptor]

@thorn forum do you pre-spawn on the client both the target and the projectile?

[thorn forum]

no, i pre spawn nothing on the client. only server it spawning entity

ill see if i can find more info on the entity with inspector egui

[spring raptor]

So you spawn a target, then projectile and then can remove the target early?
This should be fine.

[thorn forum]

between when my troop start its attack to a specific entity, and when the projectile is spawned (the animation time for example) the target could have been already killed and despawned by something else. i still spawn the projectile with the targeted entity (the entity was stored on the troop as the target).
so a entity with Projectile(target_entity) is spawned, but the target entity doesn't exist anymore.
i do this because even if the target is killed i want to finish the animation and launch the projectile, my troop don't cancel their attack. i have a history of the target position to still move the projectile.

sorry if this isn't clear lol

tldr:

so a entity with Projectile(target_entity) is spawned, but the target entity doesn't exist anymore.

[spring raptor]

Hm... Judging by the panic, the target of your projectile from which we trying to get the entity is a valid entity.

[spring raptor]

Check if such entities continue to exists on the branch when the panic is disabled.
I want to know if they are created by replicon somehow due to mappings.

[bleak sky]

Hello, I'm having issues with channels and renet when sending an event, it just panics with the message Called 'send_message' with invalid channel 2.

I've seen a Github issue and discord messages that solved the issue by adding channel setup in ConnectionConfig but I've done it already. Here is the relevant code with the logs of both server and clients https://gist.github.com/JulienLavocat/a3bfafb23fee8f1ac8546a855e2b1b06
I also tried to register the event on different channels (Reliable, Unreliable, Ordered) and the same thing happens.

I must be doing something wrong here but I don't really know what 😅

Replication was working fine, just events

[echo lion]

Can you show the code where you're adding channels to RepliconChannels?

[bleak sky]

So this might be were I messed up because I don't see what you're talking about ^^ I never add any channels to RepliconChannels 🤔

I'm not doing much more than what is done in this example:
https://github.com/projectharmonia/bevy_replicon_renet/blob/master/examples/simple_box.rs#L70

[spring raptor]

This means that channels wasn't properly configured.
If you use original renet, I suspect that you forgot to initialize client or server this way:
https://github.com/projectharmonia/bevy_replicon_renet/blob/cc83a29ab616e1086e135814679331d4ff4e0449/examples/simple_box.rs#L80

[spring raptor]

Ah, never mind, you shared the code here ^ 🤔

Looking into it

[bleak sky]

Yeah I was about to say, both client and servers a properly initialised in the gist.
But I don't think you're wrong, the logs on the gist shows that only these channels are created:

ConnectionConfig { available_bytes_per_tick: 60000, server_channels_config: [ChannelConfig { channel_id: 0, max_memory_usage_bytes: 5242880, send_type: ReliableOrdered { resend_time: 0ns } }, ChannelConfig { channel_id: 1, max_memory_usage_bytes: 5242880, send_type: Unreliable }], client_channels_config: [ChannelConfig { channel_id: 0, max_memory_usage_bytes: 5242880, send_type: ReliableOrdered { resend_time: 0ns } }, ChannelConfig { channel_id: 1, max_memory_usage_bytes: 5242880, send_type: Unreliable }] }

They don't seems to match the ones from the library

[spring raptor]

Also in your logs it says

2025-02-02T19:31:08.728468Z DEBUG bevy_replicon_renet: creating channel config `ChannelConfig { channel_id: 0, max_memory_usage_bytes: 5242880, send_type: ReliableOrdered { resend_time: 0ns } }`
2025-02-02T19:31:08.728479Z DEBUG bevy_replicon_renet: creating channel config `ChannelConfig { channel_id: 1, max_memory_usage_bytes: 5242880, send_type: Unreliable }`
2025-02-02T19:31:08.728483Z DEBUG bevy_replicon_renet: creating channel config `ChannelConfig { channel_id: 0, max_memory_usage_bytes: 5242880, send_type: ReliableOrdered { resend_time: 0ns } }`
2025-02-02T19:31:08.728485Z DEBUG bevy_replicon_renet: creating channel config `ChannelConfig { channel_id: 1, max_memory_usage_bytes: 5242880, send_type: Unreliable }`

@bleak sky, ah, it's the ordering issue

You need to register your shared part before server and client plugins.

This is why it creates only 2 replication channels for each direction, without channels for events.

[bleak sky]

I moved the app.add_plugins(SharedRepliconTypesPlugin) just after adding replicon's plugins and it works !
Thank you very much @spring raptor

[spring raptor]

@thorn forum another idea: try another messaging backend. Like quinnet, for example. Just to confirm that it's a bug in Replicon.

It might be hard to patch it for the latest replicon master... So try to run the latest replicon release with only bugfixes on top (I remember that you need one for which I forgot to draft a new patch release).
I going to draft a new release soon.

[spring raptor]

You need to write custom serialization functions, use replicate_with:
https://docs.rs/bevy_replicon/latest/bevy_replicon/core/replication/replication_rules/trait.AppRuleExt.html#tymethod.replicate_with

I don't have an example for erased-serde, though.
If you figure it out, I would appretiate if you submit it as a doc example, pretty sure users find it useful.

But it should work.

I never tried it, but it should work.

I used reflection in my project.

Same idea, you have access to the type registry in replicate_with.

We have an example for it here:
https://docs.rs/bevy_replicon/latest/bevy_replicon/prelude/trait.ClientEventAppExt.html#tymethod.add_client_event_with
It showcases an event, but the API is the same. Maybe worth writing a similar example for replicate_with.

[hexed cove]

hello, I'm looking into using replicon in a project and I don't quite understand how setting up multiple game lobbies would work. It seems like it would take a seperate process/port for each lobby?

[hexed cove]

maybe using mpsc with aeronet_replicon and managing lobbies separately?

[spring raptor]

Hi!
Yes, I would use a separate process and port for each lobby. It's just much simpler.

You can also use a single process and separate lobbies in your game logically. You won't need mpsc, just control what connected clients see with visibility settings.

[hexed cove]

wouldn't you need to expose a ton of ports? it also seems a bit complex if private lobbies are involved and ipc would be necessary for communicating who's allowed to connect to which?

[spring raptor]

True, but I think it's limiting only if your machine have a huge bandwith?
I think it's cheaper to use several smaller machines.

It's not necessary an IPC. Usually you have a matchmaking server who decides who connects to who, then it generates private keys and starts a lobby/session on a dedicated server where players can join by the key.

[hexed cove]

hmm, so passed as parameters as the process is started?

[spring raptor]

Yeah

[hexed cove]

that makes sense, thanks

[spring raptor]

I would suggest to take a look at https://github.com/UkoeHB/bevy_girk, which is maintained by one of replicon's developers.

[hexed cove]

I also was wondering if the client/server are being built into a single binary, building gui/menus might be a bit strange? the only solution I can think of is just splitting the core game logic into a plugin, and external communication handled with bevy events so menus can interact w the game

something like this

[spring raptor]

For my game I have several crates. My project_harmonia_core crate contains only game logic. And project_harmonia_ui depends on the core and provides UI.

And finally I have project_harmonia_app that is actual app.

Another way is to separate UI by features. Just put all UI into ui module and enable it only if ui feature is present.

[hexed cove]

what does the app do that the ui crate doesnt?

[spring raptor]

So yes, something like this, but also under features or in separate crates.

[spring raptor]

It contains main function and loads all plugins (core, ui and Bevy).

[hexed cove]

using features is a bit weird to me since if I want to build for UI, i likely want to have like start menus/ options/ etc, whereas building for a headless server would mean just starting the game

they would be mutually exclusive

[spring raptor]

ui and app could be united, it's up to the developer's preference.

[hexed cove]

the ui crate depends on core but doesn't load core?

[spring raptor]

Yes

Also my game doesn't have headless server. It's not a session game.

[hexed cove]

and the ui interacts with the game through bevy events?

[spring raptor]

Events, components, resources

But all logic is in the core

[hexed cove]

oh yup, so it can pull from the core dependency

that makes sense to me, I'll try to see if I can build something similar

thanks a ton for the direction

[spring raptor]

For games with dedicated server the popular choice is to have shared, client and server instead of just core crates.

[thorn forum]

hey ! i had some time to try looking at my panic issue, but couldn't replicate today as i changed location and i am now with a good connection.
i tried using a vpn to have a bad latency, but i didn't trigger the bug.
next time im on a bad connection ill re-try figuring it out, im certain its because of the bad connection. (it was a low wifi signal). changing networking backend might fix it.
thanks a lot for the help !

[spring raptor]

Bugs like this are the worse 😢
Thanks for the update!

[thorn forum]

yeah lol

ill just live like it never happened i guess 😂

[hexed cove]

is it possible to replicate resources/state or should they be updated through messages?

[spring raptor]

No, but in the next Bevy resources become entities, so it will just start working :)

[hexed cove]

oh hype, do you know where I can see the changes?

also what about state?

[spring raptor]

https://github.com/bevyengine/bevy/pull/17485

[spring raptor]

Should be an entity in the future. Then you just mark the state component for replication.

[hexed cove]

interesting, so this looks like it replaces existing resources

resources just become a special component where only a single entity can exist

[spring raptor]

Yep

[hexed cove]

how do i replicate sprites, or is there another way to create shared visible components without gizmos?

or do I insert the sprite as it's spawned, feels a little jank

[spring raptor]

You want to insert the correct sprite after the replication.
The same after scene deserialization from disk.
It would be wasteful to serialize or replicate images.

[hexed cove]

that's fair

[hexed cove]

working multiplayer with replicon and quinnet :D

[spring raptor]

Cool!
Are you working on an ant colony simulator?

[hexed cove]

Yup!

[spring raptor]

Do you replicate all their positions or just initial state and tasks?

[hexed cove]

Replicate all the positions, I'm not sure if my logic is fully deterministic

Also, I plan on implementing fog of war in the future as well

[hexed cove]

Can default visibility policy be set per component? such as if individual players have private data that needs to be replicated to specific clients, but also public world data that needs to be shared to everyone

[spring raptor]

We don't have a per-component visibility yet. But it's on my todo list.

[hexed cove]

hype, thanks for the work. building multiplayer w replicon has been a great experience so far

[hexed cove]

is it possible to only run a system in singleplayer?

[spring raptor]

Not sure if I get the question.
But you probably looking for server_or_singleplayer condition.

[hexed cove]

as an example, I want to create a player when a player joins, but also in singleplayer when the game starts. I don't want a create a player for the server so server_or_singleplayer wouldn't work

[spring raptor]

In this case you can use not(server_running) condition

[hexed cove]

that makes sense, thanks

[hexed cove]

How can I fix this?

seems to be caused by replicating a large component

Not sure if it's size or something else

[hexed cove]

oh wait this is a quinnet issue nvm

how would I set a replicon channel to ordered for a replicated component?

[hexed cove]

the error is with bevy_quinnet, but the channels are created through bevy_replicon

mutating them like this doesn't work, I assume it has to be newly opened

although with opening a new channel I'm not sure how to get replicated components to use that channel

[spring raptor]

You can't, replication relies on specific assumptions about channels.
We use 2 channels for replication, reliable and unreliable depending on the kind of change.

[spring raptor]

This means that the component is bigger then the packet size 🤔

[spring raptor]

Or quinnet have a different max size limit. Because we split updates by packet size. But we don't split entities, so if an entity is larger then the packet size, we don't split it.
Let me check...

[hexed cove]

I've tried increasing the size limit as well

through channels.default_max_bytes

I'm basically trying to replicate these large pheromone grids related to each players colony

[spring raptor]

It's different, the problem that it can't fit into a single package.

[hexed cove]

I see

and unreliable channels are necessary for the eventual consistency implementation?

[spring raptor]

Yes

As a temporary solution you can try renet2 instead. Should be an easy switch, I think it automatically splits large messages.

[echo lion]

renet2 has the same message splitting behavior as renet

[spring raptor]

She uses quinnet

[hexed cove]

Yup, the quinnet example seemed to take the least amount of boilerplate to get started

[hexed cove]

built into renet2?

[spring raptor]

Yes, when Replicon sends updates, it tries splits messages up to max packet size. But we can't split entity components, it's important for them to arrive together.
And we rely on backend to split big messages into packets. So this might be an issue on the backend side, they should do the final split.

Another potential problem is that we don't provide way for backend to configure the max message size. Maybe we hit a limit there.

I need to take a look myself. In the meanwhile, try renet or renet2. The latter might need to be update to the latest version by @echo lion.

[hexed cove]

Thanks, I'll look into it

actually it does appear to be on the quinnet roadmap

[spring raptor]

Ah, yes, the first entry is what exactly causes the problem.

But how do you have such a huge entity?

I suspect it's not a single entity.
Looking into quinnet code, they use this limit:
https://docs.rs/quinn-proto/0.11.8/quinn_proto/struct.Datagrams.html#method.max_size
But we use this constant (I took it from renet, it's hardcoded in the same way): https://github.com/projectharmonia/bevy_replicon/blob/master/src/server/replication_messages/mutate_message.rs#L227

[hexed cove]

you think it's putting multiple entities and exceeding quinnet size limits, but valid in replicon?

significantly reducing the size of the entity does actually fix it

changing from 500x500 to 5x5 grid

[spring raptor]

Yes, Replicon makes sure that components of the same entity arrive in a single message for mutations.
But looks like in quinn the maximum message size is smaller.

[spring raptor]

Do you store the whole grid on a single entity?

[hexed cove]

multiple actually

[spring raptor]

Then this is the issue ^

[hexed cove]

I could split into multiple entities, but i suspect a single grid might already be too big

[spring raptor]

Ah, then I would definitely suggest to split huge component into multiple ones

When a component changes, we resend the whole component

[hexed cove]

i do actually need the entire component resent

hmm

I decay the pheromone grid over time

[hexed cove]

lowering to just a 50x50 grid already breaks it, i think a single 500x500 rid will be too big regardless

maybe the only solution is manually replicating with events

I'll try the renet2 solution as well tomorrow

[spring raptor]

I would suggest to temporarely switch to renet or renet2.

[spring raptor]

I see ClientId in this component, for example. I don't think you want to resend it each time something like foodchanges.

[hexed cove]

I see, that makes sense

if pheromone grids + visibility grid both update every tick, it wouldn't double send right?

[spring raptor]

Yes, we send entity followed by changed components

[spring raptor]

Just to clarify, do you have only a few entities that represend 500x500? It's not 500 entities?

[hexed cove]

Nope just a single one per client, and its only visible to that client

although 500 entities works perfectly

spawning 500 ants works :>

[spring raptor]

I would highly advice to split each cell into an entity

It's way more efficient for both ECS and replicon.

I mean the inability to send a big entity is an issue on replicon side because we don't expose a way to set the max message size. I will fix it.
But sending such big entities is not a good idea in general.

[hexed cove]

all the cells are updated every tick at the same time so sending at once made sense to me

I also often need to query by grid coordinates, so I would still need a large grid stored somewhere, and possibly replicated

imagine if the player highlights a tile and I need to immediately check if that tile is empty. the player would need to query a grid somewhere as iterating through entities is jank. I'd probably end up sending the player a group of entities to refer to each cell, then query that grid + query for cell with entity

seems a lot more complex compared to updating the grid solution

[dire aurora]

This is a problem you usually see with voxel engines, every chunk is an entity, but you need to be able to immediately get surrounding chunks or all chunks a ray passes trough, for that they usually just have a resource that stores grid pos -> entity

[hexed cove]

like here right?

grid of entities

[dire aurora]

Yea, you grab the resource, then access the cell entity. You don't need to sync the grid since the entity would have a grid position which you can keep up-to-date in the grid resource trough hooks

All of this was a lot more painful before hooks tho

[hexed cove]

that makes sense but

[spring raptor]

If you update each cell every tick, then yeah 🤔

[hexed cove]

i dont think it has any significant perf improvements or helps with any usecase I have

and I want to focus my efforts on creating a playable game

I'll try the renet solution and circle back to this if needed, or maybe in the future if it helps solve a problem

[dire aurora]

I think you mentioned visibility, for which you'd want to not replicate too big of a region 🤔

But that's more chunks than "cell as entity" I think

[hexed cove]

maybe I could handle decay w timestamps instead to avoid this, but again no perf issues yet

i want to try per cell visibility for now, every ant has its own vision radius and im not sure how i'd want to chunk it

[dire aurora]

A lot of games chunk it purely for the benefit of having preloaded a few extra steps, which means you won't see things pop in because a message arrived late, while the potential harm of someone cheating remains fairly minimal

• visibility checks can get very expensive if it's more than just a radius around some point

[hexed cove]

I'll keep that in mind if it's needed, doing visibility checks per ant will probably be quite slow but we will see

[spring raptor]

Looking into quinnet, looks like this propery is set per entity.
I can provide an API for it, but it will be junky. I'll wait for resources as entities to make it work properly.

[hexed cove]

vision

700 ants starts to chug lol

[spring raptor]

Looks cool!

[spring raptor]

How do you send position, is it just Transform?

[hexed cove]

yup, i know transform can be optimized to only send translation but im lazy

[spring raptor]

Okay, I just curious how far it's possible to push it :)

[hexed cove]

well currently the grid is the bottleneck

but gonna set up renet2 tmr

also haven't built for release yet but the processing itself starts chugging my pc, the grid data doesn't change much with ants increasing

[spring raptor]

Release should be way faster

[hexed cove]

the multiplayer connection seems to be doing better than the actual ant processing

[spring raptor]

Yeah, replication is quite optimized.

[spring raptor]

Interesting, looks like https://docs.rs/bincode/latest/bincode/fn.deserialize_from.html allocates.
It uses IoReader under the hood:
https://docs.rs/bincode/latest/src/bincode/de/read.rs.html#37
We might need to stop using std::io::Cursor completely 🤔

There is deserialize_from_custom that accepts a special reader.
But I also considering swapping bincode to postcard. They are very similar, except postcard can work with no_std and have more suited varint encoding for us.

I keeping an eye on bitcode, but until they introduce streaming API it's a no-go.

[hexed cove]

is there a way to manually manage when a replication is sent?

override automatic replications w manual replications without having to use events

events work but then you have to synchronize the creation of the entity as well as update each field manually

I could also use a "real component" and "replicated component", and clone the real component into the replicated component to trigger it but that's also janky

usecase: spreading the fog of war processing over 10 ticks for performance and I don't want to replicate a half updated fog of war map

[spring raptor]

Yes, replication is send when you update the RepliconTick. Just set tick rate to manual.

[hexed cove]

perfect

oh, tick rate is set for all components, as well as updating replicontick

So I wouldn't be able to constantly update some entities, while periodically updating others

[spring raptor]

Yes, you can't replicate components with a different tick rate right now. We have a feature request for it, though.

Might get to it at some point, but I don't think it will happen soon.

[hexed cove]

can you link it?

[spring raptor]

https://github.com/projectharmonia/bevy_replicon/issues/331

[hexed cove]

i dont quite understand how replication conditions cleanly solves the usecase, in my situation I would need to trigger replication when my map is done processing. trying to set up a condition to replicate w state or something seems difficult

since the map would only be complete for a single tick before processing the next iteration

[spring raptor]

In your case the condition will be "map is done processing"

[hexed cove]

the map would be done processing for a single bevy tick, what if the sending rate is too slow to align with it?

this might be a pretty niche situation though

[hexed cove]

maybe dual component system is fine

[spring raptor]

You will send it according to tick rate, condition just tell if it's ready by this and if it's not, I won't send it.

It least that's how I planned it. I pretty sure I will provide more details when I get to it. I always discuss such features before the implementation.

[spring raptor]

It's possible workaround. Are you getting bottlenecked?

[hexed cove]

still not quite sure if I get it

[spring raptor]

I don't have a full picture either.

But the idea is to let user to decide when send mutations for specific component

[hexed cove]

a little bit, calculating the vision area per ant is quite expensive and offsetting it does actually improve release perf

[spring raptor]

It only affect mutations

[spring raptor]

Have you tried avoiding sending Transform?

[hexed cove]

not bottlenecked by replicon, just cpu processing lol

[spring raptor]

Just create Position component and do a trigger that updates Transform.

Ah, I see 🤔

[hexed cove]

the map itself is not expensive to send, but the creation of the map is

[spring raptor]

Ah, so you want to delay initial replication?

Like right after connection?

[hexed cove]

oh no i think youre misunderstanding

fog of war grid

expensive to update every frame

1. clear the current fog of war grid
2. process every n-th ant every tick
3. once grid is full rendered, replicate to client, back to step 1

so maybe an event or replicated/rendering grid system is a good enough workaround?

[spring raptor]

Could you elaborate on 1?

Not sure if I get this part.

[hexed cove]

so grid of booleans set everything to false

update the area around each ant to be true

thats it

[spring raptor]

Okay, I think I get this part. So you bottlenecked by the CPU?

[hexed cove]

Yup, and staggering my processing of ants solves this

but it leads to incomplete grids

[spring raptor]

But could you elaborate how replication is related?

Not sure if I get the connection

[hexed cove]

tick 1: clear grid, process group 1 of ants -> replicating here is problematic because half the grid isn't complete
tick 2: process group 2 of ants -> goal is to replicate here

[spring raptor]

By any chance you refer to a tick as to a frame?

[hexed cove]

yeah

[spring raptor]

So in you case the tick is each frame?

That's huge, especially for something like your use case

I would highly advice to go with a fixed tickrate

[hexed cove]

it's moreso an example

of how the ticks/frames will possibly not line up

my tickrate is fixed

[hexed cove]

i gain pretty decent perf splitting this accross like 10 ticks

[spring raptor]

Sorry, maybe it's just a bit late night for me, but I still don't get it 😅
Why do you need 2 ticks to process 2 groups of ants? Why not process all groups of ants and only then replicate?

Ah, I think I starting to get it.

[hexed cove]

this might be a pretty niche case though

[spring raptor]

It's for performance reasons, so you don't want to process all ants at once, you split then across multiple frames.

[hexed cove]

yup

[spring raptor]

How about bevy tasks?

It's just a perfect fit for it

[hexed cove]

multiple tasks for parallelization?

[spring raptor]

They designed to split hard work across multiple frames.

[spring raptor]

Not necessary, it can be a single task

Maybe a task for each group of ants will be a good starting point 🤔

[hexed cove]

can't parallelize because they need to update the same grid

but still could be good

[spring raptor]

Ah, I see

[hexed cove]

although I would still want to manually replicate once the tasks are done

[spring raptor]

Maybe replication is not a good approach for grids 🤔
I would go with a syncrhonization event.

Maybe with replication conditions it will be possible to simplify, but for now you will need an event

[hexed cove]

yup, or dual component system

[spring raptor]

Because if I understand correctly, you want the grid to be updated rarely.

And everything else updated more often

[hexed cove]

yup

[spring raptor]

Could also work, but events will be just more convenient

[spring raptor]

Is each cell changes in this scenario?