#:rainbow: Rainbow Team Relations :rainbow:

1 messages · Page 3 of 1

cosmic elk
#

OK that is not what I expected.

#

Does the documentation on bubbling behavior cover this?

runic shadow
#

Checked the impl and that's definitely how it works, not sure how clearly it's documented

#

Since entity observers are just scoped versions of regular observers any time an entity observer would trigger a global observer will also trigger

#

The bubbling is a layer on top of that so it just triggers the observers normally at each point of the traversal

cosmic elk
#

Well, that would certainly solve my problem.

#

We should be very clear about bubbling behavior in the docs.

#

There should also be a unit test which checks this, so that some future maintainer doesn't inadvertently optimize it away.

#

I'm thinking about refactoring my widgets so that they all share the same set of observers. For example, button and checkbox both have similar pointer handlers.

alpine patrol
#

This is correct

#

I originally had it the other way and rewrote it to have it literally work like the event is triggered successfully up the hierarchy. Global observers are triggered multiple times when bubbling.

#

I can work on improving the docs.

#

Aervery was also surprised by this behavior when I talked to him about it.

cosmic elk
#

I think part of the reason it's surprising is that it's not how it works in HTML, so people with long experience in the web might not expect it. However, I think this way is much more useful, because it lets us handle bubbling events globally with few observers.

runic shadow
#

Yeah, it's not like registering a listener on the window (in js land), it's more like registering a listener on everything

alpine patrol
#

but i do actually have registering listeners on the window working locally, for picking events.

#

i just need to find time to polish it and open a PR.

#

maybe tomorrow, now that i'm mostly over my cold.

cosmic elk
#

Oooh, i just realized that I could a single handler that works on all windows

#

This means my top-level FocusKeyboardEvent handler doesn't need to iterate through all windows installing an observer.

alpine patrol
#

oh that's clever. my tests were also using window observers, but actually i can just add a clause to my existing global observer for each event type.

#

i'm quite enjoying how global observers provide a new way to define shared immediate behavior for components, in the same way that systems define shared scheduled behavior.

cosmic elk
#

What this means is that I can just install add_plugin(TabNavigation) and everything will just work

alpine patrol
#

that Target<T> api really would be the cherry on top

cosmic elk
#

if we really need for the first argument to be special, I wonder if it could be a tuple.

alpine patrol
#

we should open an issue for it probably. it's a solid design, maybe someone will come along and implement it.

cosmic elk
#

I think I may have already, let me check

#

If you want to comment and sketch out what you are thinking that would be great

alpine patrol
#

will do

alpine patrol
#

i think we might be able to get away with the following pretty easily

world.observe(|trigger: Trigger<Pointer<Click>, &mut Button>| {
    let button = target.data();
    /// add behavior common to all button components
});
#

have something like Trigger<'w, E, D: QueryData = ()> and do a the archetype check and query in the observer dispatcher

#

but it conflicts with the Bundle that's already there.

cosmic elk
#

This may be a stupid idea: world.observe(trigger: (Trigger<Pointer<Click>>, &mut Button), query: ...etc - in other words, overload the In parameter.

#

Kind of like a SystemParam++

wary fjord
alpine patrol
#

You could run the query in the dispatch system and pass it in

#

It doesn’t need to be system param

wary fjord
#

stuffing it into the trigger is probably doable

#

It would have to function like a system param at least so the system knows what it accesses

#

otherwise this would cause issues

world.observe(|trigger: Trigger<Pointer<Click>, (), &mut Button>, my_query: Query<&Button>| {
   my_query.get(trigger.entity()) // UB
});
#

it'd be better to do a whole separate SystemParam, however then SystemParam needs to be able to access the input data, and I'm not sure how to work out the generics for that

#

so the alternative question is

// how do we prevent this system from getting constructed
world.observe(|trigger: Trigger<Pointer<Click>>, this_button: Target<&mut Button>, my_query: Query<&Button>| {
    my_query.get(trigger.entity())
});
runic shadow
#

Yeah that's one of the questions that made me drop it from the initial implementation, originally trigger was query like but I didn't support system params, when I added system params it became complicated

#

It can be done by adding extra validation to system construction that takes into account the in parameter

#

Or doing some refactor that means trigger is no longer the in parameter and acts like a normal system param

wary fjord
#

How would you pass the system the trigger data besides as system input?

#

put it in the world and then grab it?

runic shadow
#

Yeah it's not simple, you could write it to a component on the observer entity or something similar to sneak it in

#

But you still need a way to pass some context to the system like it's own id

wary fjord
#

I think we could turn SystemInput into a subset of SystemParam

runic shadow
#

Yeah that could work

wary fjord
#

I'll take a look

runic shadow
#

Or at least let system input share the same validation implementation

wary fjord
#

a version of validation and get_param that also takes the input data is what I'm thinking

#

3 lifetimes rust_on_fire type Item<'world, 'state, 'input>: SystemInput;

wary fjord
#

Does the data need to be mutable? An example would be helpful should be fine actually

wary fjord
#

eugh this causes issues with exclusive systems still needing to accept inputs

cosmic elk
#

Similarly, the checkbox and disclosure toggle widgets (both of which toggle on click) now share the same set of observers, because they both have a ToggleState component.

#

The ButtonState component contains the on_click callback id (which wraps a one-shot system id). Similarly, the ToggleState component contains an on_change handler, whose In parameter is the checked state of the widget.

#

This means that instead of having hundreds of observers, I have seven.

#

The benefit is not simply less memory use, but also it means that when viewing observers in the world inspector, the resulting list is much less confusing.

wary fjord
#

Honestly surprised how comparatively little is needed to add it

#

Also probably missing a few cases where param methods are called but input methods arent, will check tmrw

echo ore
#

Linked lists using custom relationships anyone?

struct LinkedList;

type Next = OneToOneDirected<LinkedList, true>;
type Previous = OneToOneDirected<LinkedList, false>;

world.register_component::<Next>();
world.register_component::<Previous>();

let a = world.spawn_empty().id();
let b = world.spawn(Previous::new(a)).id();
let c = world.spawn(Previous::new(b)).id();
let d = world.spawn(Previous::new(c)).id();

world.flush();

assert_eq!(world.get::<Next>(a), Some(&Next::new(b)));
assert_eq!(world.get::<Next>(b), Some(&Next::new(c)));
assert_eq!(world.get::<Next>(c), Some(&Next::new(d)));
assert_eq!(world.get::<Next>(d), None);

assert_eq!(world.get::<Previous>(a), None);
assert_eq!(world.get::<Previous>(b), Some(&Previous::new(a)));
assert_eq!(world.get::<Previous>(c), Some(&Previous::new(b)));
assert_eq!(world.get::<Previous>(d), Some(&Previous::new(c)));

world.entity_mut(a).despawn_recursive_with_option::<Next>(true);

assert!(world.get_entity(a).is_none());
assert!(world.get_entity(b).is_none());
assert!(world.get_entity(c).is_none());
assert!(world.get_entity(d).is_none());
#

Got directed one-to-one relationships working, not just undirected. Next is directed one-to-many, and then un/directed many-to-many

#

But many-to-many is not going to be as nice to work with, since there isn't a simple Wrapper(Entity) component you can add to one Entity to modify the relationship

rose pendant
cosmic elk
#

I should be clear, this is not how I would want to do every widget.

rose pendant
#

Yeah, that's why I said closer. I'm using a single observer per widget (component) to trigger reactivity

cosmic elk
#

Particular for app widgets that are one-offs, it's much less burdensome on the developer if all of the various aspects of the widget - appearance, state, and logic - can all be co-located together instead of being scattered across multiple source files.

#

However, for widgets that are "commodities" like button - that is, there are lot of the same kind of instances scattered around the UI - being able to centralize the logic into a single handler has more pros than cons.

#

UIs in the dot-com era were typically organized by vertical separation into layers, whereas modern UIs tend to be mostly organized by horizontal separation into functional areas.

alpine patrol
#

is there an actual write up that covers what you guys did for the render world somewhere?

unborn quail
#

the retained rendering world?

#

I think the best your going to find is the migration guide and the current docs

alpine patrol
#

if i tried to write a slightly more expanded migration guide for rendering devs, do yall think you could help me?

unborn quail
#

(not to brag but I think that the current migration guide is pretty good)

#

sure

alpine patrol
#

can you link it?

unborn quail
#

it's attached to the PR

#

let me quickly find it

alpine patrol
#

which pr

#

got it

unborn quail
#

the big thing is that you are now having to think about what entities you extract every frame and which ones just sit there

#

because entity leaks are very possible with the new approach

#

so that's important to add

#

also RenderEntity is going to implement WorldQuery so instead of querying &RenderEntity you can query RenderEntity and get the correct entity directly (no entity.id())

alpine patrol
#

i think your migration guide is great, but the render devs seem like they need a more cohesive document with specific consideration for common rendering abstractions like ExtractedInstances

unborn quail
#

sure thing

#

Write up what you think needs to be written up

#

I can check it if you want to

cloud plover
#

@spring hinge ❤️

spring hinge
#

My brain is too tired atm, taking a break from bevy ATM

unborn quail
#

I might need to take a break as well after all the retained rendering follow up is done. Part of me is trying to see if it's possible for me to get a burnout and while I love a good science experiment, I don't think I'd like it to succeed.

unborn quail
#

hey @timid mountain, just out of curiosity, I see that you have 5 Prs open (one in draft but still). Are you willing to put them up for adoption?

#

A couple are related to relations

#

Out of curiosity, have you given some thought as to how we could turn QueryState into a component and store it in the world?

timid mountain
#

Sure, any PR of mine is always up for adoption

unborn quail
#

Sure thing

azure turret
unborn quail
#

No

#

I do what I want

wary fjord
# unborn quail Out of curiosity, have you given some thought as to how we could turn QueryState...

The simplest solution is to store Entity for the state (as you did in your PR), but return Query<'w, 'w, D, F> instead of 'w, 's. That means we can still keep a reference to the QueryState in the Query. The problem I ran into and why I stopped is we need to decide where the querystate's archetypes will be updated. get_param might work but it was a bit of churn switching the &SystemMeta reference that gets passed to it into a &mut SystemMeta

#

For now I'm working on a game project to see what's the most practical and useful ecs stuff for me to work on next

wary fjord
unborn quail
wary fjord
#

I don't, sorry

unborn quail
#

All good

echo ore
#

Should probably put this here too!

cosmic elk
#

As I've previously mentioned, I would like to define an Owner/Owned relationship, such that:

  • When an owner is despawned, all of it's owned entities are automatically despawned as well (regardless of whether you use despawn or despawn_recursive).
  • Owners can also be Parents, but the set of Children and the set of Owned are separate sets - iterating over children doesn't iterate over Owned.
    "Owned" entities would be things like reactions, mutables, effects - satellite helper objects that are attached to an entity but not part of the visible hierarchy.
#

I suspect that your PR + a little bit of component hook magic would be enough to implement this.

echo ore
cosmic elk
#

It's one-to-many, you can have multiple owned for an owner

echo ore
#

The despawning behaviour is something I think you could add using required components and a hook perhaps?

cosmic elk
#

Basically an on_remove hook for Owner.

echo ore
#

Either way, the way I added these relationships is documented in a private trait Relationship that I think would be very easy to copy for adding totally custom behaviour

#

Less than 100 lines of bevy_ecs-dependent code to create the hooks for a relationship type, including events

#

Oh actually events might be the simplest answer here

#

Because this PR will fire a RelationshipEvent<R> for adding and removing

#

Where R is your custom event

viscid python
#

With retained rendering world is the only big item remaining for MVP relations archetype GC?

wary fjord
#

I'm not sure what that all requires tbh

viscid python
#

The rendering world was the main blocker for relations. With retained rendering world that blocker is gone.

wary fjord
#

Right

viscid python
#

There's archetype GC which you need with fragmenting relations because you're going to cycle through archetypes quickly & after that I can't think of anything else that's stopping an immanent MVP relations PR.

wary fjord
#

We don't need components as entities for mvp relations?

viscid python
#

IIRC no because it's just ID pairs. Components as entities just changes components to be the same ID type.

#

Also most of the hard work for making them use the same ID pool for Components as Entities has already been done a few versions ago.

wary fjord
#

we have Identifier but we're not using it in archetype storage

viscid python
#

that + (simple) queries for them would basically be MVP relations

runic shadow
#

Yep, it's just archetype clean-up that's required for relations now

#

Queries as entities would be nice to do first since we could transition to observers before making table/archetype deletion a thing, but it's fine either way

#

90% of the work left for components as entities is also just archetype clean-up, so you can get that in there pretty easily

frank hare
#

will a component relationship be a unique component ? Like can you only have one Listen<any component> or you can have one of each Listen<component_type> for each component type ?

viscid python
#

you can configure exclusivity

#

by default they'll be exclusive so you can only have one (Eats, *) for example per entity

#

you could change this to allow for multiple entity targets

quick goblet
#

What's a good place to start in understanding the state of entity relationships on main today?

#

Or design work that's been solidified.

unborn quail
#

I'd have to do some digging about the stuff that has and hasn't been implemented for the rfc

quick goblet
#

I always forget about pins, thanks.

unborn quail
#

For the minimal implementation section, we've done the first two things

#

i.e. merge observers and a retained rendering world

quick goblet
#

Queries as entities seems interesting.

#

kinda hurts my head actually.

unborn quail
#

it's not the most difficult thing in the world. Every query holds a QueryState, this PR just tries to make QueryState a component and have a Query hold an Entity reference to said QueryState instead of a QueryState directly

#

it's the details that are hard

wary fjord
#

Queries as entities is not required for MVP relations FYI

#

they would be nice to have though

#

The next step is archetype cleanup iirc

#

Although maybe even that isn't necessarily required

unborn quail
#

yes, but QueryState keeps an internal record of all archetypes that match, so if you're gonna do archetype cleanup, you should also do it there

#

I agree that we 'technically' don't need much of this stuff

wary fjord
#

We could add a remove_archetype method to SystemParam

unborn quail
#

sure

#

it's just, there's a bunch of caches (and there are going to be more) just subscribing to an 'archetype removed' event with an observer is a lot easier

wary fjord
#

true

unborn quail
#

relationships is interesting, because we technically could skip straight to implementing it, but the performance would suck and that's not really how I'd want to roll out the feature

#

it'd be one of those things you'd have to specifically warn people for, like yes we have relations but it's slow

#

I'd much prefer it if the feature got implemented and we could tell people to just go nuts with it

quick goblet
#

OK help me understand this snippet:

alice_mut
    .insert_relation::<Eats, Fruits>()
    .insert_relation::<Eats>(apple)
    .insert_relation::<Eats>(banana);

What happens if you don't call .insert_relation::<Eats, Fruits>()?

wary fjord
#

I would rather have something that works good enough sooner, than something that is near perfection much later

quick goblet
#

I'm less concerned with performance at the moment, because I'm trying to design a crate which needs the interface.

#

The general interface I'm looking for is basically just a HasMany, HasOne, and BelongsTo relationships.

unborn quail
quick goblet
unborn quail
#

it could very well

quick goblet
#

I'm somewhat surprised that the generics work out with this.

unborn quail
wary fjord
#

Like, in my mind MVP relations shouldn't have the expectation to be used in performance critical code

unborn quail
#

I agree and I'm coming around to the 'sooner rather than later idea'

quick goblet
#

I'll need to store information when I get to implementing join tables.

unborn quail
#

however, I will say that relations cause the number of components to explode and that slows down all operations if these optimizations aren't made

#

and that might not be a very big problem for users who have a decent amount of control over the amount of relations they use, but this becomes a bit of a problem for external plugin code

wary fjord
#

Pop the user-facing API into an experimental module then

unborn quail
#

imagine adding a plugin that uses relations and the whole application slows down

#

which would suck

unborn quail
quick goblet
unborn quail
#

at the end of the day, I'll review anything related to relations

wary fjord
unborn quail
#

my bad

quick goblet
#

No worries.

#

I'm just slowly collecting requirements and designs for my SQL crate.

#

The idea was to use something like SeaORM and have it map to entity relations, but I'm not sure I actually want to use SeaORM. But the idea is the same.

unborn quail
wary fjord
#

Yea I agree

#

Once 0.16 dev starts proper, I'll hop back on the ecs dev train

#

Currently taking a break besides writing my PRs' release notes

unborn quail
#

Like @timid mountain has a couple of PRs open that just changes a couple of datastructures to be able to cope with an arbituary number of components, those PRs are perfectly fine and could be merged pretty soon

unborn quail
quick goblet
#

Are entity relationships going to be bidirectional?

unborn quail
#

I believe one-directional

runic shadow
unborn quail
#

ty

quick goblet
runic shadow
#

It would add an Eats relationship between the alice entity and the Fruits entity/component

#

Seems weird to do it that way when you have entities for apple and banana, but it's allowed

wary fjord
quick goblet
runic shadow
#

Yeah you would need different methods

quick goblet
#

OK, cool. I'm not crazy.

runic shadow
#

The main blocker is archetype clean-up but the implementation of that is less the existance of a remove_archetype method and more implementing that method is very painful due to the data structures that currently track archetypes/tables

#

For queries it's not so bad but you have to deal with the accesses of systems as well

quick goblet
#

So when I do a.insert_relation::<Something>(b) which side stores the ID or reference?

runic shadow
#

That is equivalent to inserting a component on a with the id (Something, b)

quick goblet
#

OK, so it's a.insert_relation::<BelongsTo>(b) if I'm following how it works in other ORMs.

runic shadow
#

It depends what the relationship represents

quick goblet
#

Yea actually it's not exactly, because I can have many...

runic shadow
#

Inserting a relationship onto an entity is similar to a tagged foreign key

quick goblet
runic shadow
#

Where you can have multiple foreign keys for a given relationship type, since each pair is a unique component id

quick goblet
#

But I'll also need the flip side where I can iterate over the as which have a relationship with a b.

runic shadow
#

Yeah we will definitely support that, maybe not in the first PR but as a quick follow up

#

It's easier to implement the other way first

quick goblet
#

Sure

#

I'm just struggling to imagine the API.

#

Like I don't understand Targets in the hackmd.io doc.

runic shadow
#

Targets iterates the other way. i.e. if I call a.insert_relation::<ChildOf>(b) and look for Targets<ChildOf> on a I will get b

quick goblet
#

In active record I'd say a.bs or b.as if they had a join table.

runic shadow
#

You could have a similar TargettedBy<ChildOf> that would match on b to give you a, but that's harder to implement due to the way bevy queries are currently implemented

quick goblet
#

So it seems like I might want to query Targets<HasOne/Many> and iterate that.

#

When you say "you could" you mean Bevy could? Targets is part of the Bevy API for this, no?

runic shadow
#

Targets is in the MVP RFC, TargettedBy is not

quick goblet
#

So now I'm wondering how the join table case would look. Where I have two component types each with a relation to many of each other.

runic shadow
#

In order to do "proper" joins we will need to refactor queries to support matching across multiple entities, which will be complex but super powerful.

quick goblet
#

I feel like I need to read all this slowly and really think about it.

#

I'm so used to thinking of relationships based on who holds the foriegn key.

#

I guess in my model, I'd have a component declare somewhere that it has a relation to another component through some intermediary component. Whereas, with this that relation would really just be represented as a more complex query.

#

One advantage to the former is that it could allow a developer to treat direct and indirect relations the same.

#

For example:

filling.fruits = [apple, banana]
cake.filling = filling

cake.fruits => [apple, banana]
polar vortex
#

🤔 I didn't fully understand the conversation, but

  • the reason for relationships to be components is because a query finds archetypes (sets of components) that satisfy its criteria
  • the reason for the information to be encoded in the component ID is because it's nice to not need data or need to access it
    • the idea takes advantage of the fact that IDs have 64 bits yet still-living entities can be ID'd using 32 bits
      • that's only enough space to fit 2 entities, so we only get 2-ary relationships
#

The setup was designed to reuse the existing ECS machinery (e.g. components, archetypes, and an index mapping each component to the archetypes that have it).

#

the way queries (will) find archetypes is very much like prolog

#

(I think it's interesting that components and queries conceptually map to prolog's facts and rules. It suggests that queries could someday be used as aliases for tag components.)

quick goblet
#

tag components sound like a possible solution to my join table problem

polar vortex
#

by tag component I mean a zero-sized component

quick goblet
#

oh

polar vortex
#

Declaring rules like this is the equivalent of defining a query and declaring it as an alias for the component.

Purple(X) :- Red(X), Blue(X).
#

we wouldn't want to answer to "yes" to the question of "does alice have Purple?" if Purple had data because alice wouldn't have any Purple data we can reference, but there's no problem with zero-sized components

quick goblet
#

That makes sense to me.

polar vortex
#

As long as the ECS can be made aware that relation R is symmetric, then when you add (R, e1) to entity e0, we can have it automatically add (R, e0) to entity e1.

quick goblet
#

If I have a relation from A->B and another relation from B->C, I may want to iterate A's Cs.

#

Sorry, I'm getting myself mixed up. That's another thing.

#

So let me give you my actual use case since I'm confusing myself now...

I have Factions and Systems. There's a join table in my DB system_factions which also stores additional information like influence, happiness, etc.

#

I want to model this as relations, where a system has many factions, and a faction has many systems.

quick goblet
polar vortex
#

So provided you can tell the world what properties a relation has (i.e. is it transitive? is it is it reflexive? is it exclusive? etc), the ECS has a pair of methods to enforce those properties.

#

The first is that a world can explicitly instantiate components according to the rules, so adding a relationship to one entity could make the world add relationships to other entities. Pretty straightforward.

quick goblet
#

I don't follow.

polar vortex
#

Like in the symmetric example I just gave.

quick goblet
#

I think the first example I gave is the very definition of a transitive relation.

polar vortex
#

The second, more interesting one is that queries can "traverse" relationships to answer questions and return references to components on related entities.

#

So if you want to find spaceships docked on planets that are ruled by allied factions, a query would (eventually) be able to do that.

# 
SpaceShip
(Faction, $ship_faction)
(DockedTo, $planet)
Planet($planet)
(RuledBy, $planet_faction)($planet)
(AlliedWith, $planet_faction)($ship_faction)
#

Transitivity would most likely be handled this way.

quick goblet
polar vortex
#

The best example I can give is that flecs has an IsA relationship that is transitive, and if you ask if an entity has T, a query walk up the IsA chain until it finds a T or the chain stops.

quick goblet
#

You wouldn't need to write out the whole query.

#

I feel like we're kinda talking past each other.

polar vortex
#

I guess so. I don't know what all these SQL terms mean in the context of an ECS that organizes and finds entities solely based on the components they have

quick goblet
#

I'm literally just trying to model relationships where data has foreign keys to other data.

#

I think the key feature of the interface I want is to be able to setup a relationship where it traverses many relations like your example above with the spaceship, but I don't need to write out all the intermediate links because it was encoded in the relationship structure.

#

Like if I had a relation a.relates(b) and b.relates(c), I want to be able to query a.cs without thinking about b directly.

#

So I'd write a.relates(c).through(b) or something.

#

then a.cs should work.

polar vortex
#

🤔 flecs has a concept called "prefab slots" but AFAIK it's only used with the parent-child hierarchy. Like, a spaceship entity might have a whole sub-hierarchy of entities beneath it, but you just wanna call spaceship.element(turret) to jump directly to the turret entity.

quick goblet
#

Yea that doesn't seem helpful to me.

polar vortex
#

Instead of having to traverse instance_of_turret.child_of(foo), foo.child_of(bar), bar.child_of(spaceship), you just say spaceship.element(turret).

#

It's just some syntax sugar. When the entity spaceship was instantiated (along with its hierarchy), it was given a component that had the ID(turret, instance_of_turret).

#

The element method (or whatever it's called) just looks for (turret, *) in spaceship's archetype and, if it finds one, returns a ref to the instance.

#

So far, it sounds to me like you want to be able to register a similar kind of shortcut that queries can understand, instead of having to write out the full query statement.

quick goblet
#

but ideally that shortcut is tied to the definition of the relationship.

#

This is what I'm hoping to be able to model on top of the ECSs relationships: https://guides.rubyonrails.org/association_basics.html#the-types-of-associations

Ruby on Rails Guides
Active Record Associations — Ruby on Rails Guides

Active Record AssociationsThis guide covers the association features of Active Record.After reading this guide, you will know how to: Declare associations between Active Record models. Understand the various types of Active Record associations. Use the methods added to your models by creating associations.

#

I'm essentially trying to find the most natural mapping between the concepts.

polar vortex
# quick goblet basically.

well, I'm sure it's possible, but bevy kinda needs to implement query traversal first before its UX can really be improved

quick goblet
#

What is query traversal?

polar vortex
#

that complex query

quick goblet
#

ah gotcha.

polar vortex
#

answering it requires traversal

quick goblet
#

Anyway. I gtg for now. Maybe I'll try typing up a more explicit set of examples of how I might like to be able to represent these relations.

viscid python
#

it just means setting X-R->Y also sets Y-R->X
edit: oh that was explained

polar vortex
#

The tl;dr is that ECS queries will work like Prolog, so if you know how to express this stuff in Prolog, that should (eventually) translate 1:1 in Bevy.

#

All relationships will be inherently bidirectional because, like Prolog, queries will find arguments that satisfy predicates through unification.

#

To give an example, this Prolog query would yield all (parent, child) pairs.

// Parent and Child are variables
?- ChildOf(Parent, Child).
#

If you set Parent to "James" and leave Child as a variable, it'll yield all ("James", child) pairs, i.e. it'll find all of James' children.

#

If you set Child to "Alice" and leave Parent as a variable, it'll find Alice's parents.

#

When I went through those associations, this is what I got:

  • belongs_to => seems to have the same properties that "child of" has
  • has_one / has_many => a relation R can be marked as exclusive, so attempting to add a (R, bar) will replace any existing (R, foo)
  • has_*:through => You would achieve this with a complex query. If you want to shorten it, I think the natural expression would be Bevy supporting "rules" (queries serving as aliases for components).
#

For an example of a rule, I'll use one of the has_many:through examples from the Rails link you shared. In Prolog, you could find a doctor's patients (or a patient's doctors) using this Patient rule.

Patient(X, Y) :- Appointment(X, Y), Physician(X).
#

When the Prolog interpreter later sees Patient(X, Joy) used in another query, it'll (basically) auto-expand the head (Patient(X, Joy)) and replace it with its body (Appointment(X, Joy) and Physician(X)).

#

(recursively if Appointment or Physician are also rules)

#

In ECS terms, rules would be queries that you can treat like components, either as new components or as aliases for existing (zero-sized) components.

#

e.g. this complex query

SpaceShip($this),
Faction($this, $ship_faction),
DockedTo($this, $planet),
Planet($planet),
RuledBy($planet, $planet_faction),
AlliedWith($ship_faction, $planet_faction).

can be shortened to this

Spaceship($this),
AlliedPlanetFaction($this, $planet_faction).

by defining a query that functions as an AlliedPlanetFaction relation and has the rest of the terms

AlliedPlanetFaction(X, Y) :-
Faction(X, $ship_faction),
DockedTo(X, $planet),
Planet($planet),
RuledBy($planet, $planet_faction),
AlliedWith($ship_faction, Y).
#

(it's good to be familiar with it since it's where this stuff is heading, IIRC flecs is 99% of the way there, just doesn't have the rule support yet)

quick goblet
#

has_* is the inverse of belongs_to so the fact that relations will be bidirectional means that, IIUC, there will really only be has_*.

#

I'm not quite sure how exclusive relations work. Seems like they should only apply in the has_one case, and could hopefully allow for a query without iteration. Whereas has_many is the more general concept, where you'd iterate the related components like normal.

left summit
fossil basin
#

An important difference is datalog is not turning complete so it’s guaranteed to terminate. Seems desirable for a query language

craggy venture
#

datalog and prolog are used for somewhat different purposes
datalog is more suited for databases
and then there's https://github.com/vmware/differential-datalog
something like differential datalog is used for the in-progress new rustc trait solver because the semantics line up nicely

polar vortex
#

I'm not saying we should use Prolog as a literal DSL, only that it's a natural way to model ECS queries

polar vortex
ancient spear
#

Is there any way we could shove query variables into the type level api?

polar vortex
#

idk, probably not, strong-typing is pretty antithetical to relationships because entities don't exist at compile-time

#

you'd only be able to express a limited subset of what's possible at runtime

ancient spear
#

I just mean like the complex flecs query you posted above

polar vortex
#

That's why we now have a query builder API. There will eventually be a system builder API. You can already define new components at runtime without type information. You just can't express everything with types alone. So IMO it's not rare, we just don't have all the tools to support it yet.

quick goblet
polar vortex
ancient spear
polar vortex
#

@ancient spear, like I know Bevy is super Rust-centric, but like... releasing an editor is only gonna increase demand for some official kind of scripting (like blueprints or Lua or something).

#

I think as t goes to ∞, less code will be statically compiled, so more code will be able to use the "full power" of ECS.

ancient spear
polar vortex
#

Yeah, I worded that poorly, sorry. I meant to state it like "I do think it'll be awkward since Bevy built a culture around Rust and strong-typing, but I think that's likely to fade into the background in the future. And there will be less of a need for type system wrangling."

#

MVP will probably not include all the fancy query traversal stuff.

ancient spear
polar vortex
#

Like, I think it'll just start off as "you can query for relationships" and it'll be a bit before query variables and finding things by following relations.

ancient spear
alpine patrol
polar vortex
#

Rust will still be the foundation, but like fewer and fewer people will deal with it directly, probably.

ancient spear
#

yeah probably

#

I would just hope that doesn’t lead to the rust API being any less really really good than it already is

#

Not that I’m worried since there’s very little momentum in that direction as-is

#

I don’t mean to sound super assertive either, things will evolve fine I think :)

polar vortex
#

it shouldn't get worse

#

I think things will converge to: There's a feature-complete "dynamic" API (written in Rust). Both the strongly-typed Rust API and any scripting interfaces bind/wrap those "dynamic" data structures and methods.

#

like, everything will be the same underneath

#

i.e. there will only be a single form of a system and it won't matter if its body is a Rust function, an FFI function, or a function that runs a runtime-compiled visual script

#

I think that's what it means to be data-driven.

#

The ECS itself has always been a runtime thing, it's just wrapped in a strongly-typed interface. Components are just blobs of data with a known layout (and methods via reflection).

#

Strongly-typed code just knows that information upfront without asking. Scripted code would have to get it through reflection.

alpine patrol
#

I think we will always have a larger set of users focused on the rust api.

#

Adding scripting would be more about expanding to new users.

#

Think engine devs vs technically-focused game designers.

polar vortex
#

what I mean is that adding support for scripting should look nothing like a new language, it's just a new wrapper over the same operations

ancient spear
#

For the ecs specifically yeah

polar vortex
#

like, there wouldn't be "rust queries" and "lua queries" (that'd be silly), whether a query is created through one API or another, it should produce the exact same struct for the ECS to consume

#

(just to reiterate, there's an abstract ECS model / "dynamic" API that doesn't involve a type system, the strongly-typed Rust API is layered on top of it)

polar vortex
# ancient spear For the ecs specifically yeah

Right, but there's nothing but entities, components, queries, and systems. Adding new features just means adding more of those. Users aren't supposed to invent layers upon layers of abstractions like they often do with a traditional, object-oriented engine.

ancient spear
#

nothing is stopping scripting from using the abstractions themselves though, but it might be a little more verbose

polar vortex
#

A query is just a machine that takes in a set of component IDs and spits out a set of archetype IDs that have those components. No type information is involved.

#

The strong-typing just sandwiches the actual work. T is used to acquire a TypeId to lookup a component ID on one end, and to safely deref the returned pointer as a &T or &mut T on the other end.

ancient spear
#

yep

polar vortex
#

The ECS could (and should) have a runtime iterator struct that strongly-typed Rust iterator APIs just wrap.

#

Scripting languages wouldn't bind the strongly-typed Rust methods, they'd bind the underlying ones (i.e. get_by_id instead of get). We're just still missing that clean separation in a few key places (queries, systems, commands).

#

Like, this is how flecs can have C#, C++, Rust bindings. Those are all wrapping the same underlying C data structures and calling the same underlying C functions.

#

Traits are an interesting point, but I believe they should ultimately be handled the same way. The ECS should introduce a canonical v-table representation (it already kinda has one, see ComponentDescriptor) and components should create them when they're registered to a world. Like, reflection should be part of the ECS model.

ancient spear
polar vortex
#

I'm getting kinda rambly, but ultimately all I'm trying to say is that the Rust experience should continue to improve even if scripting does blow up in popularity. All future language bindings should naturally look and feel similar since they'd all be wrapping the same stuff the Rust API wraps.

frank hare
#

will relationship allow to iterate by a "runtime value" so the complexity = O(n) where n = the amount of entity with that "runtime value".
Ex: iterate entity with component A that are owned by the player A "with the Owned component with a relationship to the player A"

viscid python
#

The complexity will be the same as queries ie O(n)

#

except you'll have back tracking because of acceleration structures like the component index so that's a pretty fast n

#

And queries can be cached which will save you from doing the lookup on subsequent runs when there were no structural changes

frank hare
noble panther
#

probably meant i.e.

frank hare
#

i don't even know what this abbreviation mean

noble panther
#

"same as queries, in other words O(n)"

#

i.e. is "id est", meaning "that is"

frank hare
#

amazing

viscid python
#

oh nvm I've been believing a lie for half my life

frank hare
#

must have been confusing a lot of people during that time

viscid python
#

no cause in essence has an equivalent meaning that makes more sense cause it's not old english or whatever id est is

frank hare
#

i just want to iterate over the cards with x components of a specific board that hold them

viscid python
#

yeah that's a very fast operation & something relations are good at

frank hare
#

do it change the archetype or just hold the component index of the entity of that have x relation value of their relation ship

#

i guess the second

viscid python
#

changes archetypes

frank hare
#

oh, i'm fine with this

#

i'm always iterating for cards in the same board anyway

noble panther
viscid python
#

ah yes ofc it's latin

azure turret
#

and I was so sure it was "?? example"

#

I guess twitter had a character limit also back when people spoke latin

steep veldt
viscid python
#

Foo(X), Bar(X) :- Bazz(X).

#

if I remember my prolog correctly

#

global predicates composed of compound facts (or other rules) that procedurally determine other facts

steep veldt
viscid python
cosmic elk
#

@high stone Is this correct?

#[require(Node, TextLayout, TextFont, TextColor, TextNodeFlags)]
pub struct Text(pub String);

I thought that Text was supposed to represent a single span? I'm having trouble constructing paragraphs because it's trying to lay out each span as a separate node.

viscid python
cosmic elk
#

Ignore me

viscid python
steep veldt
polar vortex
polar vortex
#

A "rule" infers a truth from other facts and rules.

// Two people are friends if they have something they both like.
friends(X, Y) :- likes(X, Z), likes(Y, Z).
#

In ECS lingo, a fact is having a component.

#

Rules would be queries that you can use as component terms in other queries.

#

Like, rules are about inference.

#

Rules don't really add (edit: much) new power to queries, but they really help to shorten complex queries. The part on the left of the :- is just shorthand for the part on the right.

#

Does it make sense? Like, referencing queries as terms in other queries to reduce boilerplate.

#

🤔 well actually I guess being able to create rules that serve as aliases for other facts and rules does count as a new power.

polar vortex
#

If you ask Prolog if the friends(bevy, flecs) predicate is true, it'll check if that fact exists in the knowledgebase. If not, it'll try binding bevy and flecs as the arguments for every 2-ary friends rule that exists, one-by-one, until it finds one that unifies (is true) or it runs out of rules.

steep veldt
#

let me try to code your example 🤔 I'm curious

polar vortex
#

I'm pretty sure I haven't misrepresented what flecs is or isn't capable of.

steep veldt
polar vortex
#

🤔 query scopes is probably the closest thing atm.

#

I brought up rules earlier because someone asked about, like, creating shortcuts to use in queries later. Like, this is a pretty long query (taken from here).

SpaceShip($spaceship), 
Faction($spaceship, $spaceship_faction),
DockedTo($spaceship, $planet),
Planet($planet),
RuledBy($planet, $planet_faction),
AlliedWith($spaceship_faction, $planet_faction)
#

An author might wish to factor out some of these terms into something they can use in other queries without having to type them all out, like this:

Spaceship($spaceship),
AlliedPlanetFaction($spaceship, $planet_faction)
#

where AlliedPlanetFaction isn't an actual component, it's just a rule.

AlliedPlanetFaction(X, Y) :-
Faction(X, $ship_faction),
DockedTo(X, $planet),
Planet($planet),
RuledBy($planet, $planet_faction),
AlliedWith($ship_faction, Y).
steep veldt
#

this is what I build, probably not as simple as stating facts, rules like you said:

# 
#[test]
fn test_facts() {
    #[derive(Debug, Component)]
    struct Friends;

    #[derive(Debug, Component)]
    struct Likes;

    #[derive(Debug, Component)]
    struct Boardgames;

    let world = World::new();

    // if Alice Friends Bob, then Bob Friends Alice automatically
    world.component::<Friends>().add_trait::<flecs::Symmetric>();

    let query_boardgame_lovers = world.query::<()>().with::<(Likes, Boardgames)>().build();

    world
        .observer::<flecs::OnAdd, ()>()
        .with::<(Likes, Boardgames)>()
        .each_entity(move |e, _| {
            let e_id = e.id();
            query_boardgame_lovers.each_entity(|boardgame_lover, _| {
                if e_id != boardgame_lover {
                    //this makes e friends with boardgame_lover as well
                    boardgame_lover.add_first::<Friends>(e);
                }
            });
        });

    let alice = world.entity_named("Alice");
    let bob = world.entity_named("Bob");

    alice.add::<(Likes, Boardgames)>();
    bob.add::<(Likes, Boardgames)>();

    //observer making them friends
    println!("alice Friends bob: {:?}", alice.has_first::<Friends>(bob));
    println!("bob Friends alice: {:?}", bob.has_first::<Friends>(alice));

    let query_fact = world
        .query::<()>()
        .with_first_name::<Friends>("$Y")
        .set_src_name("$X")
        .with::<(Likes, Boardgames)>()
        .set_src_name("$X")
        .with::<(Likes, Boardgames)>()
        .set_src_name("$Y")
        .build()
        .each_iter(|it, i, _| {
            println!(
                "entity who are friends and like boardgames: {:?}",
                it.src(i as i8).name()
            );
        });
}
# 
---- test_with_trait stdout ----
alice Friends bob: true
bob Friends alice: true
entity who are friends and like boardgames: "Alice"
entity who are friends and like boardgames: "Bob"
polar vortex
#

the example I gave was kinda the opposite

#

let's say we have this Friends relation

#

but Alice and Bob don't literally have it

steep veldt
polar vortex
#

Alice doesn't have Friends(Bob) and Bob doesn't have Friends(Alice)

#

But if there's a rule like Friends(X, Y) :- Likes(Z, X), Likes(Z, Y) , then if Alice and Bob both had Likes(Soccer), the ECS would infer that they are friends. A query on Friends(Bob, *) would yield Alice even though they don't actually have the literal relation.

#

IIRC Sander was building toward supporting rules, but rules are not just sub-queries.

steep veldt
#

A query on Friends(Bob, *) would yield Alice even though they don't actually have the literal relation.
Yeah I see, that level of magic isn't there yet indeed kek
you could ask a query "are bob and alice friends" if they both like boardgames, but that's the extend of it

steep veldt
#

if you have a friends component

#

add a query attached to the friends component

#

and then in the core code, if Bob has no literal friends relationship, check if friends has queries attached to it, execute and yield the results

#

this probably wouldnt work for all cases, but in this case, it might

polar vortex
#

You'd want it to be automatic / built-in into the query engine. Like, yeah, you'd define a query and tie it to the Friends component. But to count as a true rule, every query that had aFriends term in it would need to automatically pick it up.

steep veldt
#

yup

#

thanks for the explanation~

tight salmon
#

We have psuedo Rules already in impl'ing WorldQuery, no?

polar vortex
#

If we look at a rule like

// If X satisfies a and b, then it also satisfies c.
c(X) :- a(X), b(X).

If a, b, and c where compile-time components then, sure, you could impl a new WorldQuery as a shortcut for Or<(With<C>, (With<A>, With<B>))>.

#

But what sets rules apart is the whole "produce new information" thing. Rule composition can be much more flexible. Like, they can be endlessly and recursively composed.

// If X is Z's parent and Z is Y's parent, then X is Y's grandparent.
grandparent(X, Y) :- parent(X, Z), parent(Z, Y).

// A parent is also an ancestor.
ancestor(X, Y) :- parent(X, Y).
// A parent's ancestor is also the child's ancestor. 
ancestor(X, Y) :- parent(Z, Y), ancestor(X, Z).
#

I should say I think bevy is pretty far off from thinking about rule support. It's definitely outside the scope of this working group IMO. I only brought them up to explain how that Ruby on Rails stuff could be translated to the ECS.

rose pendant
rose pendant
#

Ideally it should work as if Bob has (Friends, Alice), but what if Friends was a rule with >2 arguments. Then what does Bob have

#

Been thinking a lot about combining this + the ability to represent pairs/facts as entities

#

So Bob could have (Friends, Alice, Besties) where Alice, Besties (or Friends, Alice) is represented by a single entity

#

Still pretty far away from being able to do inference though. The brute force way would be to just run all rules for each entity but that's not feasible

#

Feels like it should be possible to look at the terms of a rule (and nested rules) and find which components contribute to it

#

So you could eliminate running that rule for all archetypes that don't have the component

#

(but that gets complicated quickly with operators, multi-source queries, traversal etc)

polar vortex
slate quartz
#

Hey, I think working relations would allow for exploring/unblocking some better ui ideas, and also be useful on my personal project.
What are the current blockers/ is there something I could help with to make this happen faster?

unborn quail
#

there are @slate quartz, you could technically already implement it, but they big 'blocker' we currently have is the handling or archetype cleanup

#

how familiar are you with the ecs internals?

slate quartz
#

probably not as familiar as I should be for something like this.
I understand the general terms, but I'd probably need to look at the source code to have a concrete idea.
I'm familiar with how ecs's work in principal at least.
For anything specialized I'd by happy to learn if you happen to have some links handy.

wary fjord
#

next step in that post for us is cleanup

unborn quail
wary fjord
#

although we aren't following it exactly (we skipped over components as entities)

unborn quail
#

currently the various datastructures rely on the fact that

  1. there won't be that many archetypes
  2. we won't ever remove them (or need to) because 1
#

so the goal currently is to adapt the datastructures that deal with archetypes to they handle 1) removing arechtypes (cleanup) and 2) a generally larger amount of them

slate quartz
#

Ah okay; has there been any progress on that front/is there a draft pr/fork of it wip right now somewhere?

#

(or is it still conceptual phase)

unborn quail
#

the first two are relatively easy to adopt, the third one is slightly harder and the last one is very very difficult

#

the trickiest thing is that most of these are all performance sensitive so you need to do some benchmarking

cloud plover
twilit grotto
#

Hey 🙂 had a read through that roadmap, and I'm curious, are relationships planned to be able to store data eventually? It's not on the roadmap (at least not explicitly) so I would assume it would happen after all those items, if at all

alpine patrol
#

I believe so? Guess I always assumed, since they are represented by rust types.

viscid python
unborn quail
twilit grotto
#

Fantastic 🙂

rough gyro
#

Yo folks! I've been a bit active in this space, perhaps a year ago, but have been on a long hiatus since. However, since the closing of pull #15635, I think I want to get back into it a bit. I feel like that pull got closed not so much because of the approach or the extra features, but because there is no clear consensus on how the next step for the API for relations should look, especially in terms of consistency, regardless of the underlying implementation of those relations.

So, I propose the following:

  • I'll start asking questions one by one to flesh out the important bits of the API and consistency model
  • A hackmd is created to iterate on the design together, probably based on the contents of rfc pull #78 / #18
  • Decisions for which there is consensus are recorded down so we don't need to revisit them in each pull

While we don't want to write an implementation that is impossible to implement, I'd rather not want to go into fragmenting / non-fragmenting relations and the whole implementation plan. Instead the first step is understanding the API and how it interacts with reflection, scene save / load, direct mutations, etc. And even though the API design should go further, I'm looking for something that's realistically compatible with the Parent/Children components that are in use today, or defines a path on how users will move from those components to whatever API we wish to have for this thing.

And just to avoid misunderstandings, I don't want to dictate how the API is or how to do this thing, I merely want to offer my help in facilitating the discussion and moving realistically forward - if my help is accepted.

Making sure the peeps I know that have been heavily part of this in the past see this: @cloud plover @unborn quail @polar vortex @viscid python

cloud plover
cloud plover
viscid python
#

In separate PRs.

cloud plover
#

That work is generally valuable

#

But we need an actual plan, and a single source of truth we can point contributors to

viscid python
cloud plover
#

Sure, go ahead and implement it. But there's no reason to block design work.

rough gyro
#

Like said, I want talk about the API, how it looks – any implementation work is independent of that

viscid python
echo ore
cloud plover
#

And in particular, what level of guarantees we want to provide

viscid python
rough gyro
#

I'm not at all opposed to progressing in the background for fragmenting relations or just uncontroversial work items related to this - as long as every step is an improvement over status quo, it's just a positive and shouldn't be blocked

#

If Parent & Children becoming private is transparent for the user, then I think it's uncontroversial – if it requires a migration guide entry, then I think we should probably be aligned on if we want to hide those or not 🙂

viscid python
#

There's like a 1 line migration that's about it.

#

The stuff you could do before like directly replacing components would invalidate hierarchies so it's no longer allowed.

cloud plover
#

And fully agree on splitting it out into small PRs!

viscid python
rough gyro
#

I personally believe we should also align this with https://github.com/bevyengine/bevy/discussions/14437 - how are relations written in bsn! syntax? How will they show up in scene files? If we decide all the relevant components are private, then it will also impact that side – otherwise it's possible to create invalid hierarchies by manually writing a scene that contains the wrong components.

GitHub
Bevy's Next Generation Scene/UI System · bevyengine bevy · Discussi...

Preamble My vision for Bevy UI is largely a vision for better Bevy Scenes and an improved Bevy data model. The general angle is: Embrace the "Bevy data model" for both Scenes and UI. Fill...

# 
let p = foo.take::<Parent>().unwrap();
bar.insert(p);

I consider this absolutely awful in the first place 🙂 But I don't know if people actually write code like that.

viscid python
#

They don't which is all the more reason I'm fine with breaking it

cloud plover
#

Yeah, I'm very okay with users not bieng able to manually remove the Parent / Child component

viscid python
echo ore
cloud plover
#

ParentInserter is definitely not going to fly IMO

rough gyro
# 
- e.get::<Parent>()
+ e.get_components::<Parent>()

Quick question about this... if I have an EntityRef, can I call get_components on it directly? Or do I need some Query separately?

viscid python
#

Ref is fine

cloud plover
cloud plover
rough gyro
cloud plover
#

But it does get a lot harder with relations

viscid python
cloud plover
#

Yeah, I agree with that 🙂

#

We need to have a validation step during scene parsing

#

(which is important to write down)

#

And I would like to work with #1264881140007702558 to figure out syntax for it

rough gyro
#

Yeah, there's a gradient:

  1. relations are never out of sync
  2. relations are only out of sync if using manually edited scenes
  3. relations are out of sync if mutating components manually, but not if replacing them whole
  4. relations are out of sync if not using the blessed API but touching components manually
#

Now we are at 4, I think 🙂

viscid python
#

Also while sure planning helps get more contributor bandwidth planning for this is a bit premature. It is likely other parts of bevy will not work with BSN immediately & it will be easier to design for when BSN is actually released.

rough gyro
#

Well, the big question in my mind is: the next generation scene/ui system seems to capitalize on components at the primary concept that needs to be understood, not bundles or other things. that there's a simple mental model that covers everything and not multiple separate models. So, if relations are not components, doesn't that mean that there's two models to understand – components and relations? Is that a good direction and also blessed by cart?

viscid python
cloud plover
rough gyro
viscid python
#

So, if relations are not components, doesn't that mean that there's two models to understand – components and relations?
Relations are components.

#

They're going to require a new notation but they're still components.

cloud plover
#

(this is why I would really like a central user-facing design doc)

alpine patrol
# rough gyro Well, the big question in my mind is: the next generation scene/ui system seems ...

if i had to hazard a guess, i think bsn will go through several rounds of iteration once it's in active use, especially as it relates to the identification of entities within the scene document. once we get that nailed down, i would generally expect that the hierarchy continues to work as expected (even when the hierarchy is relationified) and that additional relations/links can be created within a scene as normal.

#

theres other stuff, like relations between scenes, and currently i sort of expect we won't even try to support that

#

all conjecture on my part

viscid python
# cloud plover (this is why I would really like a central user-facing design doc)

I'm just not sure what that's supposed to do? What would I be designing? Every aspect of the relations API? Most of "how would X look in bevy" is just "look at flecs we're copying that" & the other questions like "how's it going to play with bsn"... bsn doesn't exist yet so you're asking to design how it'd behave with something that doesn't exist? It's a dsl anyway we can define whatever we want. Like how detailed are you imagining this doc to be? Do you just want imagined rust & bsn versions of the examples in the flecs manual? Cause a massive plan outlining everything is not feesable or productive.

cloud plover
#

Most of "how would X look in bevy" is just "look at flecs we're copying that"
I'm a maintainer and SME-ECS. This is not what we're doing.

alpine patrol
cloud plover
#

You're welcome to not participate in the process of "define what the API should look like", but I would appreciate if you stopped discouraging others from pursuing valuable work.

rough gyro
#

I sincerely hope we are not copying Flecs to do entity.add(Parent, other) as "add" is totally unintuitive at least to me. But I have no opposition on the general idea of looking like the Flecs API, except different where we want it to be different.

cloud plover
#

Yeah, flecs is a really valuable source of prior art, and unless we have a good reason, I'd like to align where possible

#

But "just copy flecs" isn't the plan that @crude oracle or I have in mind, and we've publicly said so in the past

rough gyro
#

But, I think this discussion does make it pretty clear we need to have a consensus on how entities look and behave from the user point of view – as people are not aligned on that. I'm not so concerned about the Query API as more query stuff can always be added, but the other stuff is important, regardless of the implementation.

crude oracle
#

Plenty of great things to learn from in Flecs, and I do suspect a fair amount of overlap.

#

But we should consider each design decision on its own and pursue whatever makes the most sense for Bevy

viscid python
# cloud plover > Most of "how would X look in bevy" is just "look at flecs we're copying that" ...

Terminology will be changed ofc but I'd be quite suss if it didn't look very similar to flecs especially the query builders. I've seen this tendency to make things "bevy" which often feels like designs that're trying to be clever with the kind of patterns bevy devs are used to only for it to be reverted into something simpler a few releases later because the design turned out to be impractical.

cloud plover
alpine patrol
cloud plover
#

And in order to actually plan that out and evaluate it, we need to write it down.

viscid python
alpine patrol
#

i think it's good to have 'design pressure' in favor of both accepted solutions and exploration, it's part of how we self-moderate. it seems like alice and cart reacted poorly to "just copy flecs" because that would be favoring accepted solutions in excess.

#

there's a middle ground of exploration and borrowing, and part of defining that approach is (as alice said) to write down our choices.

#

to document when we are choosing to follow flecs, and when not, and the reasons why

cloud plover
#

I'm also particularly frustrated by @viscid python speaking authoritatively without any granted authority. It's confusing and discouraging. I really value your expertise, but your tone often comes across in a way that shuts down conversations and presents consensus when there is none.

#

Even as a maintainer and SME, it's important to distinguish between "I think it would be good if" and "we have decided that"

viscid python
# cloud plover I'm also particularly frustrated by <@66149552897396736> speaking authoritativel...

Sorry if I come across as such. I have no authority here & I've made every effort to be careful & prefix my own thoughts with "I" & welcome if you'd point out anything I missed. I'm not seeing where I went wrong rereading my messages. I'd also like to say I find assertions of "That work is generally valuable" without any explanation frustrating. Cause I have reasons why I think certain things are not valuable (atleast not now) with often the only consensus I can see being:

  • Vibes
  • We don't like where we are now & want something different
rough gyro
#

BTW, just for clarity: when I say "we", I mean an inclusive we, as in all of us together - not "we" as in some specific group of people to which I belong and some others do not. It's just a figure of speech for me.

cloud plover
#

I'd really like to record both the what should we do and the "why should we do it" in a single place though.

#

I agree that our design and decision-making in this working group in particular often feels pretty vibes-based, without the appropriate context or rationale for anyone to really feel confident in the choices made.

#

I can lay out why I think it's important for us to prioritize writing a design doc focused on user-facing API and terminology if you'd like?

viscid python
# cloud plover I broadly agree with your critiques of the linked PR, FWIW. Your tone is often v...

I was discouraging some of the work in the PR but in the case of the design doc I really don't understand it & I'm trying to understand why it's wanted. Cause this has been asked for multiple times, has been produced in a couple of different iterations, with each time not changing much about peoples understanding? I do not see what I currently understand as a "design doc" being helpful for that reason. I believe just getting features out there is much more helpful to people's understanding. Like I recall lots of discussions with a negative reception when it came to one shot systems & observers even for very clear use cases like ui & the editor. That completely changed when the features landed. Just getting the features in peoples hands contributes to their understanding far more than API designs because API designs ime are more easily understood by people already familiar with the subject matter. I think as more pieces of this come into place consensus will be easier to reach which is why I think this is unproductive.

cosmic elk
#

What would be helpful is something more like a traditional TDD, one which explains the problem, gives background and terminology, and presents a solution in an organized manner.

cloud plover
#

I have a few reasons why I'm in favor of it:

  1. New contributors constantly come and ask "how can I help with relations", and there's no good onboarding material to point them to.
  2. This can evolve into the module docs for the user once it's published.
  3. I have a hard time tracking the whole set of features in my head, and want an overview to chart out problem areas.
  4. I want to be able to define a user-facing API so we can get a crude version in place early in the process, and then iteratively refine the backend without breaking users.
rough gyro
#

Well, for one – @echo ore writes a pull request which preserves consistency on all operations except component mutation. And to be fair, the ask was to convert current Parent/Children into using component hooks which explicitly do not support catching mutation. And many people seemed fine that the consistency was enforced unless kind of intentionally trying to break it – but some people like @viscid python were not, saying that consistency should be maintained also for that, meaning that the components need to be private. And it's not clear what people are actually expecting on that front - what is acceptable. By documenting these assumptions, we get these decisions written out and then pull requests will not be rejected based on requirements that nobody has written out or that haven't been generally agreed upon.

runic shadow
#

IMO even if you take the internal implementation of relations as a black box there is still a huge design space that needs to be explored for the API and how to communicate it to users. Questions like: How do you build queries for them? How to add and remove them? etc. etc. which the MVP RFC doesn't cover exhaustively at all. Having that discussion now can be productive immediately, of course it needs to take internals in to account to some degree. flecs definitely doesn't offer all the answers here, even for the basic stuff that we can assume to be shared (with, without etc.) it's API makes heavy use of operator overloading that don't directly map to rust code. The rust binding can offer inspiration but it's definitely imperfect.

You also need to worry about making sure users actually understand how to use the API in a way that they can apply it to their projects.

cloud plover
#

I also don't want the implementation PRs to get bogged down with design discussion around APIs, dragging out the process 🙂

#

Those are often the most contentious!

#

It would be great to have a consensus API already designed before the PRs are even opened, and then we can just check for correctness during code review

cosmic elk
#

You can add appropriate caveats saying "this is not the final API"

runic shadow
#

As far as the implementation is concerned there are two main routes to relations:

  1. Something like the implementation in the PR above that's built on top of the current ECS features. This gets it in to the hand of users quickly and we can get some common use cases established. Although I do share @viscid python 's concern of adding all this code and then having to rip it out later.
  2. Built more deeply into the ECS where relationship edges split archetypes so that queries for them are more efficient and we can build more bespoke support for them. The problem with this is all the technical blockers which I attempt to cover in the RFC (and a lot of which are now resolved). In short we need to support world.unregister_component(component_id) since when a relationship edge is removed it's like deleting a component and that's really hard, especially refactoring bevy as it exists today.

The amount of API surface that bevy_ecs currently support makes these PRs difficult, it's very repetitive and makes the whole thing less approachable to new contributors. The main thing stopping me from taking a weekend or two and trying to implement the operation is it sounds like a huge chore and if I'm going to write boring code I may as well do it for work 😦

cloud plover
#

Yeah, the API duplication in bevy_ecs is quite rough 😦

cosmic elk
cloud plover
#

If we want to do 1, I'd really like to make sure that there's basically no migration work for users

#

The linked PR was definitely more of a detour: it introduced a number of concepts and migrating would require breaking your code

#

But I don't think that has to be true

#

I would 110% take slow relations with almost no features as a user though

viscid python
runic shadow
#

I think we can probably create an API that is easy to migrate if we design with that in mind

rough gyro
#

I agree that the linked PR was definitely a detour, but the important distincition there is – do we want to support ent.insert(Parent::new(other)) as an API? If we do that, and it behaves correctly, that's something that's really hard to take away from users once it is out there.

cosmic elk
#

To extend the metaphor further, the only way to avoid making a detour is to have a clear idea of the destination 🙂

runic shadow
cloud plover
rough gyro
#

These are exactly the decisions we need to write down – because the author of the pull certainly thought it'd be good to allow this for the user, and that's exactly what using component hooks enables.

runic shadow
#

I'm pretty sure you could implement an API similar to the one proposed in the RFC inefficiently now and there would be no migration

viscid python
#

I'll concede a bit on the design docs cause I think I've been assuming too much about what people imagine from reading flecs examples but I think any API design docs should be

  • small
  • focus on relations (ie. not getting into other things like bsn)
  • focus on immediate features of relations (basic query functionality)
cloud plover
#

Yep, totally fine with that as a scope

rough gyro
#

If we have special syntax for every relation operation, then the only place where hooks are needed is despawn - everything else can happen with the special syntax.

cloud plover
#

I want something in place that's "less bad than the existing stuff" and "future-compatible"

wary fjord
#

(IMO, and maybe this is already assumed) We should also make sure the design doc is compelling and detailed enough that users don't need to know anything about flecs to grok it

#

Keeping everything in the frame of how bevy uses and implements relations makes it easier for new contributors to get started, and not just get told "go look at flecs first"

rough gyro
#

Personally, while I think the design doc is important for a wide variety of users - the thing I consider most important is writing down the design decisions. Without a proper record of the decisions made, and the non-decisions made, it's impossible to write pulls for the implementation of things as the design discussion starts up in the pull request.

unborn quail
rough gyro
#

This isn't a big distinction, but I'm envisioning that as a working mode, in the beginning, there's a section at the end of the document that says "Design decisions" and in there there's "Question: Should user be able to insert Parent component manually? Decision: No, this is not allowed, they should use the add_relation API." etc.

cloud plover
#

Let's move this subconversation to #ecs-dev?

unborn quail
#

do you think that API surface is unavoidable alice, or are there maybe some usefull cuts we could make?

wary fjord
rough gyro
#

But, overall on this topic, I'm going to let things simmer for a while so everybody gets to give their opinion – but if the originally proposed hackmd method of working on this is suitable, could someone create a document under @bevy where we could start the work? Initial contents could be the current RFC that's on relations.

#

Or we could have a separate RFC simply for "relations API"

alpine patrol
#

I can do this when I get home this evening

cloud plover
#

The Github RFC UI is really bad for long-winded conversations 🥲

rough gyro
#

Yeah, I meant that we'd do a hackmd that is for a yet to be proposed new RFC which is only "relations API" and doesn't contain any discussion for the implementation side – and once it's close to finish, it would be migrated to Github.

#

As opposed to continuing the RFC that is currently there on the hackmd side. I'm fine either way.

cloud plover
#

And if we design nonsense we can trust the implementation folks to yell at us for giving them an impossible spec

rough gyro
#

Okay, I can try to write a skeleton for it, copying some stuff from the old one, and then we can start going the questions

#

Now for something different -question: In flecs, when trying to find the children of a certain entity, how are all the archetypes which contain the entity as a ChildOf looked up? What kind of a data structure contains the archetypes? Assuming there's thousands of objects that have at least one child, there's thousands of archetypes in fragmenting relations, so how many memory lookups does it take to get to the archetype (or multiple archetypes) that has the children in a vec?

runic shadow
#

You do a lookup in the component index (which we have) and you get a list of archetypes

rough gyro
#

What kind of data structure is the component index?

runic shadow
#

Hashmap, but we should also investigate low id optimisations

#

(wouldn't be relevant for ChildOf though)

rough gyro
#

So it's a hashmap from component id to vec<archetype id> and then based on archetype id, we need to look from Archetypes which has vec<archetype> the corret row which contains an archetype which has vecs for all the components?

runic shadow
#

The archetype contains a vec of the component ids but it doesn't contain the storage, that's either in an associated table or a sparse set

rough gyro
#

right, yeah, left out that part of the indirection

viscid python
rough gyro
#

Just to be clear, does flecs use a hash map there, or do you just mean that bevy's component index is a hash map?

runic shadow
#

Bevy and flecs both use a hashmap there, flecs has low id optimisations

#

(so for low ids it's just an array)

rough gyro
viscid python
rough gyro
# viscid python I'm not sure what it back tracks from. I *think* it's the query's matched archet...

Well, we are talking about the parent query here, so (ChildOf, *) archetype list matches every distinct parent so there'd be thousands of archetypes. Obviously the other components might filter the list, but the archetypes for a specific parent must be looked up in the component index, because every other list that contains those has a ton of archetypes. It can possibly skip the archetype ids it knows do not contain some other component needed by the query, but that's a separate part of the filtering I'm not so interested in.

viscid python
rough gyro
# viscid python Oh wait yeah mixed those up. Component index look up should be quicker because `...

Well, it gives you the archetype ids, and looking up the archetypes from the archetype vectors gives the storages and the storages then have all the components for the children, including an Entity vec which is equivalent to the Children component on Bevy. But for flecs, looking up the entity ids of each of the children isn't really necessary, unless they are also a parent and the query is recursive. Just stating out the obvious to ensure my understanding is correct.

#

Right, you edited it to say the same thing pretty much.

viscid python
rough gyro
#

One more question: when a new (ChildOf, e) is added on an entity - does it allocate a new archetype id for that if that's the first time (ChildOf, e) has appeared for a specific e? Are the archetype ids allocated sequentially, and then the archetype is appended to a linear archetype vec?

viscid python
#

Which arechtype vec are you referring to?

rough gyro
#

A global vec of all archetypes, if such exists for flecs as well.

#

In bevy it's Archetypes and it contains Vec<Archetype> and archetype id is an index into that vec. If I understand correctly.

viscid python
#

It'll make a new archetype anyway. Not sure about flec's archetype storage.

rough gyro
#

Not 100% conclusive, but:

    /* Tables */
    ecs_sparse_t tables;             /* sparse<table_id, ecs_table_t> */

    /* Table lookup by hash */
    ecs_hashmap_t table_map;         /* hashmap<ecs_type_t, ecs_table_t*> */

So, I'd say that the storage of archetypes in flecs is sparse<>, but that the component index doesn't point to archetype ids (table ids) but instead directly to the archetypes (tables) via pointers.

rose pendant
rose pendant
#

You're of course welcome to rediscover how to do these things efficiently :p - it took me years to iterate & arrive at my current design

#

The only area where I do see things diverge is the user facing API because like @runic shadow mentioned the flecs API heavily relies on features that aren't available in Rust

viscid python
#

people can complain about .set::<Parent>(e) not being "bevy" but as someone who's done all sorts of type magic with APIs before I don't see any way to make implementing .insert(C) & .insert((C, e)) not be a complete pain in the ass

rose pendant
#

For example, the current Bevy hierarchy implementation makes it trivial to store an ordered list for a parent's children which is something you can rely on today. That'll get a lot harder with fragmenting relationships

steep veldt
steep veldt
# runic shadow IMO even if you take the internal implementation of relations as a black box the...

even for the basic stuff that we can assume to be shared (with, without etc.) it's API makes heavy use of operator overloading that don't directly map to rust code. The rust binding can offer inspiration but it's definitely imperfect.
If the operator overloading is what I think it is you're talking about, rust overloads it too

.with::<Position>() is set to default .inout_none
.with::<&Position>() is set to .in()
.with::<&mut Position>() is set to .inout() IIRC

but yeah, the rust API is definitely imperfect, since it reflects for the most part on the CPP API , which does a bunch of function overloading and has stronger generics / specialization, where the cpp API has one .get and .get_mut function, rust flecs has 10 of them, or so in total, but there's no way around it.

#

preface: I have no idea how bevy versioning works

just my 50 cents reading everyone's concerns, what if you cut all the crap what if's of the API and concerns of breaking the existing API out of the window, and instead of trying to aim for a 0.16 or 0.17 release (which a lot of people do, hope for, and aim for), aim for a major 0.20 Bevy release which introduces breaking API changes to the ECS to support it's ongoing development, such as relationships. I imagine BSN will be part of that as well.

This means though you'd have a separate 0.20 bevy_ecs fork, that needs to stay in sync with PR's / fixes to the main branch, but in my eyes, would avoid a lot of wasted time fighting the current API & would allow for faster commits, iterations, alpha/beta testing the API.

viscid python
#

That sounds like you're suggesting break everything in a big release? Rust did that after it introduced editions and quickly learned that while people don't mind breaking changes they don't mind small breaking changes. Bevy wouldn't have a backwards compatibility mechanism either like editions which would make it even more painful.

steep veldt
#

not break everything, break what's necessary. Similar to how Flecs 3 became Flecs 4. In the end, the changes and migration, won't be as bad as you think, I think.

#

but the goal is simply to stop bikeshedding about the API, and start developing it

#

and once developed, you can revisit the API to align it closer to the current API where possible, if possible.

#

but since this working group existed, the goal was always "next release!" for many people, and I think, that's where a lot of the problems come from because you are required to develop with the current API.

viscid python
#

It makes development require more energy tho because you have a separate fork with all this stuff. It doesn't help testing either because people won't be able to use any of the plugins their game uses. Same problem as tracking main.

viscid python
steep veldt
#

yeah there wasn't, but I cannot count the amount of times I've read "I want to land minimal relationships in 0.xx" where xx is the next release, and similarly "I want to contribute to relationships for next bevy release" and countless times "hoping on min relationships next bevy release"

viscid python
#

I've been reading those for over 2 years at this point I pay them no mind ferrisCluelesser

steep veldt
#

it just complicates things, and if you set a deadline target, which allows more flexibility and separate development, it'll become so much easier in the end

#

develop + change API -> re-evaluate -> develop more -> refine API (trying to match current API as much as possible)

wary fjord
#

Does Sander work on flecs full time?

viscid python
steep veldt
#

instead of develop + current API constraint + delay + delay + delay (/s)

viscid python
steep veldt
# wary fjord Does Sander work on flecs full time?

Sort of.

Beforehand it was mostly his free time, but recently in his new job, working for Meta (indirectly), which uses Flecs, he's contributing to Flecs during work as well I imagine. I'm jealous, he works with the infamous Mike Acton.

viscid python
#

oh meta uses flecs?

#

damn

steep veldt
#

yes

wary fjord
#

an experimental module could probably work though

viscid python
#

Are there any releases with experimental modules currently?

wary fjord
#

text might be getting one for 0.15, I don't know the current state of that currently though

steep veldt
wary fjord
#

don't all those have full paid teams?

echo ore
cloud plover
viscid python
steep veldt
cloud plover
viscid python
cloud plover
echo ore
viscid python
cloud plover
#

Okay, I'm glad to hear it

#

But yeah, ordered children is non-negotiable for moving over our main hierarchy

viscid python
echo ore
#

It seems to me as an outsider that a lot of precursor work has to be done before user-facing features become available. Where I honestly think we should be doing the reverse; offer a relationship API that isn't performant, isn't pretty, but does offer the functionality required. Then fragmenting relationships etc. can come in and make that feature set massively better with a concrete set of benchmarks and user stories to work against

steep veldt
cloud plover
steep veldt
#

and I don't know about you, but personally, I hate admin work, I just want to code, I procrastinate admin work till I have to. That includes writing a fictional draft user API.

steep veldt
viscid python
echo ore
cloud plover
echo ore
#

(But feel free to correct me if that summary isn't accurate!)

cloud plover
#

The existing hierarchy code lives in a totally seperate crate and is just glued on: it's not slowing down development in any way

steep veldt
#

I see I see 😄 well it was all just a suggestion from an outsider lurking in, where I think having a separate branch would be more beneficial than directly PR'ing into main for relationships

cloud plover
#

Long-lived branches are really painful to review and keep up to date

steep veldt
#

(working groups solve some of that pain)

cloud plover
#

Yep, definitely some

echo ore
#

I think Bevy's current approach of doing the long-lived stuff outside Bevy as a plugin and then upstreaming once it's refined/accepted is a good approach

cloud plover
#

We're not really getting stalled out on admin work or anything: we're just missing a final desired API and there's a few tricky backend PRs to go to make it performant

steep veldt
#

like someone suggested, don't care "directly" about making it performant on first iteration

echo ore
steep veldt
#

I think even if you do, you'd still end up 10x slower than current flecs approach (or not, who knows)

cloud plover
#

I don't even care if it's slower than our current implementation

#

(but that will probably block moving Parent/Children over)

steep veldt
#

hell, your API draft should include TDD's for a mock API in practice

#

you'll quickly realize that the draft needs adjustments once you do (which happens to me a lot)

cosmic elk
cloud plover
#

But UI very much needs order-preserving

#

I'm not sure that normal game world entities do though 🤔

#

It would definitely be annoying to have the order not be stable in the inspector though. You'd at least want to stably sort them

echo ore
cloud plover
#

Yeah, I think we want query sorting at rest anyways for perf reasons

#

Which we can use to implement an alternate relations storage (cribbing from @rough gyro here, full disclosure)

viscid python
#

Yeah was gonna say that should address most ui sorting needs (atleast all the ones I can think of)

cloud plover
#

You could also add a lateral Sibling relation

#

Which would be kind of cursed

#

But open up some neat relational querying

steep veldt
cloud plover
#

(I'm increasingly convinced that only UI needs this for Parent/Child)

cloud plover
#

Elements of a todo list would be stored as children of a single parent for instance

#

And we want to ensure that they're laid out in a consistent order

steep veldt
#

yeah just seems like a "simple" query sort on inserted time, or name

viscid python
steep veldt
#

this is how the API on flecs's rust side looks like for sorting

cloud plover
#

The hope is just that we can avoid adding a ZIndex-flavored component on UI nodes that stores their relative position in the sibling list

#

Since that's a pain to maintain

#

And then you need to pull in the data every time you want to iterate the children in a stable order

steep veldt
cloud plover
viscid python
viscid python
steep veldt
cloud plover
#

Insertion time won't work well (even using change ticks) if they're inserted in the same system

cloud plover
viscid python
#

Yeah I should sleep too. If it's still unclear please tell 🙏

rose pendant
# cloud plover (I'm increasingly convinced that only UI needs this for Parent/Child)

Fwiw the current implementation of Bevy is also not ideal for UI IMO, because it's a simple vector of child ids which is expensive when doing things like removing a single child from a long list. Storage that's optimized for UI will I think look very differently from plain fragmenting relationships or bevy's Children, and will be more informed by the structure of (bsn) templates

rose pendant
steep veldt
# cloud plover Insertion time won't work well (even using change ticks) if they're inserted in ...

just made a mock todo list in flecs, with sorting (which isn't ideal as Sander pointed out, I'd probably rewrite it with a custom cache on the todo list entity that orders on insertion, removing, sort fn changed). Used atomic for the insertion to preserve order. It isn't the best way to go about it either, but simplest solution.

edit: why am I using world time even, the atomic counter would be enough, lol.

-+-+- first run : no custom ordering - use inserted time -+-+-
Todo list: "todo_list"

  - "Buy milk"
  - "Buy eggs"
  - "Buy bread"

-+-+- second run : custom ordering -+-+-
Todo list: "todo_list"

  - "Buy bread"
  - "Buy milk"
  - "Buy eggs"
📎 message.rs
viscid python
#

If you opt out of table reordering the perf should be pretty similar to how we currently sort child nodes.

rough gyro
#

As for the actual content of the RFC, I plan on starting to ask questions here bit by bit

#

I intend to put it as "publicly" editable so anyone with an account can edit, but I don't want to do that before we move it over to ensure the move is as easy as possible

cloud plover
frank hare
#

any target release version for relationship ? just so i can plan my game developement better

cloud plover
alpine patrol
cloud plover
cloud plover
#

@rough gyro done

frank hare
alpine patrol
frank hare
#

i have to know to know to decide how to plan my dev time

cloud plover
frank hare
#

ok thanks

cloud plover
#

It'll take us a long time to get up to full feature parity with flecs though 😅

frank hare
#

i just need something to query over a "value" relation ship

rose pendant
cloud plover
viscid python
rough gyro
alpine patrol
#

I also tend to configure the docs so that anyone can edit them, even without an account. Don't think normal users can do that for their docs, but i'm happy to set it up if people want it.

rough gyro
#

I've now removed my old document, so nobody accidentally stumbles upon that anymore, and restored the formatting on the current document

cloud plover
rough gyro
#

But just realized that the first thing to do is to map the current hierarchy API and the expectations around it. For example that Children is a gettable component that has mutation apis like swap, sort_by, etc. - but the way to access the entities is that it implements Deref<Target=[Entity]>. After that's "fully" mapped, we can have the discussion on backwards compatibility that which features are important to keep and which can replaced with something totally different.

#

I'll try to do this work first, before going to the actual questions to be resolved

cloud plover
#

Like the fact that you can mem swap it 😦

alpine patrol
#

Currently it’s quite difficult to get the list of ancestors for a given entity. If relations could let us efficiently query and cache that info, it would open up tremendous opportunities for optimization in various tree traversal systems.

viscid python
alpine patrol
#

Being able to enforce an acyclic graph by construction would also be huge for those same class of algorithms. They are currently unsafe because of the potential for cycles fks with threading.

viscid python
alpine patrol
#

Hmm. We can move the unsafe traversal into the ECS though, right?

viscid python
#

Wym?

#

What is "unsafe" traversal here?

alpine patrol
#

currently transform and view propagation use unsafe recursive code

#

will relations let us move the unsafe protion of that into the ecs and expose a general safe traversal api?

cloud plover
alpine patrol
#

oho lemme see

cloud plover
alpine patrol
#

hey this is ghost nodes again, no?

cloud plover
#

Nope, this is the ghost-node free version

alpine patrol
#

the gift that keeps on giving.

cloud plover
#

Actually the "iter ancestors" method already existed

#

But, still!

viscid python
# alpine patrol will relations let us move the unsafe protion of that into the ecs and expose a ...

Yes. With the new query APIs multiple entities can be touched at once during iteration. For example:

for (transform, parent_transform) in world
    .query::<(&mut Transform, &mut Transform)>()
    .with::<Parent>("$p")
    .term::<1>(|term| term.src("$p"))
    .build()
    .iter_mut()
{
    // ..
}

Can violate mut alias rules if an entity sets itself as its parent (we currently disallow this). So the thing that has to be checked for queries is do their terms ever stomp on them selves. This can also happen in a 3 entity cyclic graph where you get the second transform from your grandparent instead of your parent where you grandparent is you. This shouldn't be terrible tho? Cause the complexity should remain a constant k for any query because its terms will be fixed.

echo ore
rough gyro
#

Hmmh, maybe I should make all these lists based on 0.15 instead of 0.14 🙂

viscid python
echo ore
cloud plover
#

Is it painfully expensive if we do it incrementally?

#

Being able to assert that a relation graph is acyclic would be super useful

echo ore
#

And again, this is about correctness. Performance is a secondary concern

#

Important, but it doesn't matter how performant it is if it doesn't do what it needs to

alpine patrol
somber girder
somber girder
viscid python
#

@echo ore@cloud plover The things mentioned do not solve the problem. Even if you enforce 1 edge type to be acyclic queries can ask for multiple which means you get cycles again depending on how the terms traverse edges. You would have to check all edge types when new edges are crated which can mean going through your entire world. Not only is that impractically expensive but you also make many use cases invalid because it's almost guaranteed that cycles across different edge types will form. This is why this is best checked lazily by queries (which is still a hard problem). I apologize if I'm being overly assertive but this is just the halting problem & it's best to give up on trying to solve this eagerly.

echo ore
echo ore
viscid python
#

Even if you enforce 1 edge type to be acyclic queries can ask for multiple which means you get cycles again

cloud plover
#

I only care about ensuring that each individual edge type is acylic though?

alpine patrol
#

i think it's vitally important to allow parallel acyclic traversal on a single relation type. if this is not possible, then relations will provide very little value for me and probably won't be useful for transform propagation.

cloud plover
#

The motivation here is to allow us to kick the unsafe code out of bevy_transform, and allow users to parallely iterate over the transform hierarchy

alpine patrol
#

ideally add safe to parallel acyclic but that's a secondary concern to correct

viscid python
rough gyro
#

Okay, am I missing something stupid – VisitEntities is implemented for Parent, but not for Children, where as VisitEntitiesMut is implemented for both Parent and Children?

alpine patrol
rough gyro
#

Does VisitEntities for Children automatically work due to IntoIterator<Item=&Entity>?

viscid python
cloud plover
#

Or why it's undesirable / not valuable to enforce rules about "a single graph type is acyclic"

echo ore
alpine patrol
#

sander has told me that we can cause individual relations to be topologically sorted, surely that imples that individual relations can be made acyclic

cloud plover
#

If we had the ability to express this natively with relations, it makes the systems-as-entities migration even easier

somber girder
#

I think it might be worth separating acyclic graphs as an invariant vs graphs in general
"I wish for the ECS to maintain this graph in an acyclic state when I mutate it" vs "just give me a graph, I do not care about cycles"

echo ore
#

Which if we're making the underlying storage private, we can enforce on mutation, which means we only need to:

  • Entity is mutated
  • Traverse up the parents, adding each to a hash set
  • if a duplicate is found, cyclic! Panic! Abort!
  • If no parent is ever found, not cyclic, all good. Let's go!
echo ore
somber girder
#

If we maintain a graph that is acyclic, we can then rely on f.e. children not having duplicates, and we no longer need hash sets to parallel traverse mutably

cloud plover
echo ore
#

Heck, you don't even need the hashset, just keep going til you hit your initial entity again.

viscid python
somber girder
echo ore
cloud plover
#

And only expose the dangerous methods in cases that we've validated as being safe

alpine patrol
cloud plover
#

Getting errors on insertion makes things so much easier to debug

alpine patrol
#

and i would like it to be acyclic please

echo ore
#

"In my game, I'll never make a cycle, because I'm perfect"

somber girder
viscid python
wary fjord
cloud plover
viscid python
#

Like I'm actually mad. I'm gonna leave this discussion.

cloud plover
#

I was sincerely attempting to understand, FWIW, and have spent quite a bit of time trying to understand your concerns and see if there's a viable alternate strategy here

#

I can definitely see why it's hard to maintain acyclic guarantees for queries involving multiple relation types

alpine patrol
#

i will cop to not really understanding relations very well, but I have a very specific nail and I want to make sure we are getting the right hammer for it.

echo ore
# viscid python I genuinely feel like people aren't even trying to

I apologize, I am trying to understand your perspective here. I believe you're suggesting making cycle detection the responsibility of the Query rather than an (optional) invariant of a relationship edge type. I'm saying that I believe it is a stronger assertion to have it in the relationship itself, and that it can be done in a performant manner.

rose pendant
#

I believe it is a stronger assertion to have it in the relationship itself
this is what flecs does fwiw- though the actual detection of cycles is kind of on a best effort basis, since even doing perfect cycle detection in debug mode would slow things down by too much

somber girder
echo ore
cloud plover
#

And then you could have an unsafe method that transmutes it in some form, like assert_acyclic

echo ore
#

Yeah seems reasonable, add(parent) and add_assume_acyclic(parent)

rough gyro
#

I don't know how queries are meant to interact with relations in the current plans, but I think it's clear that if multiple relations are involved in the same query, there's no guarantees that there are no duplicate entities even if everything is acyclic. For example, if there's relations Mother and Father, and we live in Alabama, there's no guarantees that e -> Mother -> Father and e -> Father -> Father are two distinct entities.

somber girder
#

Thinking about it more, a HashSet for maintaining the uniqueness invariant is likely not avoidable, but it could be lazily updated/skipped for some operations
(In that there are transformations that preserve/do not touch uniqueness, and thus don't need the check)

rough gyro
#

Personally, I think acyclicity is just one consistency rule that may need to be maintained for some relations, but it's certainly not the only one. So I'm more thinking like component hooks, that relations should have hooks on insertion / replace that can do the necessary checks to maintain consistency. And Parent/Children may have a check for cycles. But, there's many ways to design this.

echo ore
cloud plover
echo ore
cloud plover
#

"Tree structure" for rexample

cloud plover
#

Same as we do for dense/sparse storage

echo ore
#

I reckon we need a Relationship trait with those items. Especially since we might want to enforce that the component is a ZST

cloud plover
#

Yep, probably Relationship: Component

#

As Joy would remind us though, it's important that the ECS itself is the source of truth for this data, not the type system

#

Because otherwise it breaks dynamic representations

echo ore
#

That is a good point. I guess the Relationship trait could provide object-safe methods for getting its invariants?

rough gyro
#

Personally, I didn't hate the proposal in the earlier pull which just had PhantomData<fn(&R)>. Any type could be the marker for the relation and wouldn't need to be a ZST, but it was also obvious that the it's not actually contained in the relation. So I'm not yet sold on enforcing Relation to be a separate trait like Component. But we'll see.

echo ore
#

To allow dyn Relationship (or a dyn ObjectSafeRelation)

cloud plover
#

Kind of orthogonal, but a useful desiderata

echo ore
#

Threw a section onto the hack document including that we would like invariants to be available for relationships, with some examples of common invariants

rose pendant
cloud plover
#

We absolutely can't call them "component traits" in Bevy 😂

rose pendant
#

I know, but same idea

cloud plover
#

Yeah 🙂

#

And this is a really nice list of useful behavior to consider

rose pendant
#

The acyclic/traversable traits are what's preventing applications from accidentally creating queries that can end up in infinite loops

viscid python
# echo ore I apologize, I am trying to understand your perspective here. I believe you're s...

I believe you're suggesting making cycle detection the responsibility of the Query
All of my suggestions have been just this. As was already mentioned they're not mutually exclusive with edge properties they'd just be what does the best effort checking.

We don't just have to worry about infinite loops we have extra problems flecs does not because of rust's mut aliasing rules

If a query asks for &mut Transform from both e1 & e2 here e1 -Foo-> * -Bar-> * -Bazz-> e2 there can be a cycle that can only be checked by the query
which is all the more reason to have queries check edge invariants like cycles instead of trying to have a command/hook/whatever check them eagerly on edge mutations. Simpler invariants like exclusivity, symmetry or cleanup can be upheld by hooks & observers.

echo ore
# viscid python > I believe you're suggesting making cycle detection the responsibility of the `...

I agree, the Query will need to check these things. My reason for saying we still want to include the option for acyclic invariance at the relationship level is that Query isn't the only way to traverse this data. E.g., getting the Parent from an entity using EntityRef. Having the invariant available at the relationship level also allows the Query to rely on it for its own validation. E.g., instead of needing to validate all relationship joins, it can skip validation on single-edge joins where the edge is acyclic, since by those invariants e1 -Parent-> e2 will always produce two unique entities e1 and e2.

#

From a debugging perspective as well, enforcing the invariant in the relationship means you're more likely to detect and log/panic when the invariant is broken, rather than when it is noticed

polar vortex
#

Yeah, I think "invariant" is a strong word.

#

IMO, at best, query traversal can notice cycles during whatever kind of traversal it's doing. (Enforcing cleanup polices would give more opportunities to detect them.)

#

We definitely wouldn't want to eagerly run the SCC algorithm every single time the relationship gets added to something.

echo ore
#

You wouldn't need to run the SCC algorithm for every change. Since relationship mutation would only be possible via a highly controlled API, we can run a single up-traversal starting at the mutated entity, and halting when either we hit a root, or ourselves (cycle detected)

#

I do agree with making it a feature flag so it can be disabled for maximum performance, but even included it shouldn't make much of a difference, especially with some appropriate caching

#

Also would only need to check on adding a relationship, since removing cannot create a cycle

tight salmon
echo ore
#

Since the check would be run on each mutation, you know that the previous state was already cycle free

#

Meaning the only cycle that could exist must include the current entity

polar vortex
#

Because it's way harder to statically prove that complex queries can't match the same archetype without being ultra pessimistic.

#

In a single query, each source variable contributes a set of matching archetypes. And you often can't statically guarantee multiple variables won't get the same value, so there's nCk subsets of terms that have the same source.

# 
// Just think about what sets of archetypes this can match.
A, B($this, $a), C($a), D($a, $b), E($b)
#

And to check if two queries are disjoint, I believe you'd need to test each subset in Q1 against every subset in Q2.

#

And that's kind of the easier problem. The harder problem is that complex queries can potentially alias themselves.

# 
// What if $this == $x?
T, R($this, $x), T($x)
#

So IMO what we should do is give each archetype-component a ref-counter for query terms to increment/decrement so Bevy can runtime borrowchk and error/panic instead of aliasing.

rose pendant
#

A big challenge with this is that you need really fine grained runtime checking, or you get arbitrary runtime panics

polar vortex
#

Like, it would suck to not let users construct queries like this one (or worse, require unsafe to construct them). IMO it's better to use the ref-counters to prevent aliasing, along with an escape hatch—an iterator that yields pointers instead of references (the iterator would be safe but user would have to invoke unsafe to deref).

rose pendant
polar vortex
#

🤔 to me, more panics sound preferable over paying to borrow-check component values individually

#

that's why I'd have the iterator as a backup

rose pendant
#

What I mean is that this runtime borrowcheck needs to be per component instance, not column

tight salmon
polar vortex
#

I think I understood you properly. Per-column sounds preferable to me than per-instance atm.

rose pendant
#

Yeah, but the issue with per column is that you get really arbitrary failures

#

Because if a query happens to match two entities that at that moment in time are in the same table, you'd get a panic

#

Even though the query is not really aliasing

#

And it's really easy to end up in that situation

Likes($x, $y), Likes($y, $x), Name($x), Name($y)
polar vortex
#

right, and that'll be annoying, but pushing people to reach for the "unsafe iterator" in those cases is a viable escape hatch

rose pendant
#

If x and y are in the same table, query will panic

rose pendant
#

What you're suggesting with an unsafe iterator is essentially the same as what flecs rust is doing

#

If you want to make it 100% safe you either have to compromise big on performance, or give up on a lot of useful features

polar vortex
#

I had been thinking about how to statically prove that two complex queries don't alias and about requiring unsafe to use a query that could alias itself and I had to agree back then that it would scare off people from using these features, so panics with an escape hatch just seemed better.

tight salmon
polar vortex
#

some entities like themselves, so it would borrow the same components twice

rose pendant
tight salmon
#

Oh duh, sorry i misread

rose pendant
#

If an entity likes itself you actually would get a valid aliasing issue

#

The problem I wanted to illustrate here is that it's not at all inconceivable that two different entities for x and y are in the same table

#

Which would cause a runtime panic if you do column only safety checks

#

Even though there's not strictly speaking an aliasing issue

polar vortex
tight salmon
#

Hmm would an 'Arc at home' be cheaper/less expensive then just using Arc thonk

runic shadow
#

In that example, and I think many of the common cases, you could determine during initialisation that the two sources that access the same components are $x and $y and add one equality check to the query plan. Not free, but much cheaper than locks

polar vortex
rose pendant
#

It wouldn't be 100% safe still since you can obtain references through other APIs

#

But it would provide better checking for queries yeah

runic shadow
#

Well in bevy that's already blocked in most cases

polar vortex
runic shadow
#

You can customise the behaviour, warn panic or skip

#

(I was sketching this out while imagining how I would re-write the bevy query engine to support variables and such)

rose pendant
runic shadow
#

Bevy already checks at initialisation if two queries in the same system overlap so yeah

polar vortex
#

I think the biggest issue is actually the *_many(_mut) methods

#

although I guess those would be covered by the runtime borrowchk

runic shadow
#

Yeah those methods generally pay some overhead to check for aliasing anyway, although it isn't simple

somber girder
rose pendant
#

That sounds feasible. Cool idea

#

Easy to implement as well

#

It would have a measurable perf impact since a query wouldn't be able to return entities in bulk- or would need an operation that does the alias check in bulk

#

But probably a lot faster/more storage efficient than per component instance checking

#

And you only need it for queries that actually can alias, which is not going to be the majority

spring hinge
# cloud plover Meshlets and ghost nodes

I think TAA is still technically experimental, because it still has some ghosting/sharpness issues we haven't fixed. We'll never be able to fix absolutely all artifacts, but we can do better then we have.

rose pendant
#

But I guess that's the price for satisfying the borrowchecker

runic shadow
#

Yeah, as far as rust is concerned the alternative is more UB

polar vortex
#

rust orz

#

I mean, the database will return the correct results, it's just not always safe to deref all terms at once

runic shadow
#

(and you can easily have a function during initialisation that is marked unsafe that bypasses adding the additional checks/instructions)

rose pendant
#

Won't this cause any issues in practice? iirc in some of our discussions someone mentioned that the rust compiler makes assumptions on refs not being aliased

runic shadow
#

It does, the likelihood of actual miscomplications in a case like this is low, but if you are writing and reading from both refs during the system you could get some wacky behaviour

rose pendant
#

Okay, so that's very much a you're on your own option

runic shadow
#

Yeah, you should really only use that when you know there are going to be no actual aliases and you just want to omit the checking

rose pendant
#

Right, so this would cause issues if the above query does match an entity that likes itself

polar vortex
#

what do we mean by issues?

#

bevy can't have UB, so there would have to be runtime panics to use the standard API, but yeah, users would be on their own if they used an unsafe iterator escape hatch

runic shadow
#

Yep, we are only talking about the potential for issues after opting in to unsafe

#

Unfortunately there is no safe way to allow them to do what they want (alias in a query) without changing something about what is passed to the system

rose pendant
#

Makes you wonder whether a scripting language binding for bevy should allow for this

#

It'd be nice to be able to not have to worry about these limitations in a scripting environment

polar vortex
#

the scripting environment would have to bind the iterator to allow aliasing, so rust would be satisfied as some binding would be calling unsafe

runic shadow
#

That's interesting to think about, because in theory if you pass raw pointers that alias to the scripts environment it's not like the script is going to miscompile by assuming noalias

#

Technically should be fine to skip the checks for scripts

tight salmon
#

Depending on the impl of the binding you may not need to

polar vortex
#

idk if y'all are focusing on compiler stuff, but isn't the language the ECS is written in irrelevant to whether or not a scripting environment should do anything about aliasing?

runic shadow
#

Agreed, the ECS doesn't really care what happens once it's passed off the pointers

#

It's just a little strange that the query results could be different in a scripting environment vs from rust

rose pendant
#

It's a little strange that a query returns incorrect results period :p The results of the scripting binding would actually be correct

#

Strangeness will exist with or without scripting

polar vortex
#

incorrect with the equality check thing, right?