bevy_mod_picking: the upstreamening

[summer talon]

Right now I know that pointer events are broken when dragging things onto/off the canvas. Fix in the works. Please do shoot any other bug reports my way, I'm sure there are bound to be some.

But generally, that's lovely to hear. Thanks for the feedback!

[empty iron]

Cross-posting Ord violation panic in the sprite backend, if I may: #1038322714320052304 message

Edit: I can make an issue, but I just don't know if this is a user-land issue.

[flat sky]

Hmmm yeah go ahead and make an issue either way. I'd be tempted to just wrap it in a FloatOrd and call it a day

I'll make a pr tomorrow if I remember

[empty iron]

https://github.com/bevyengine/bevy/issues/15258

[summer talon]

@viscid blaze no pressure, but did you end up taking a look at bevy_mod_raycast?

[wintry spoke]

Previously, I was getting a Pointer<Click> for everything under the mouse when clicked that has Pickable. My things are should_block_lower=false. I want to get clicks for all the things, because I need to do my own selection sorting.

With the recent change to depth, it seems like I only get a click for the topmost thing. Was this intended?

[stoic flicker]

No, open an issue please

[summer talon]

I’ll look at this asap. I suspect I know what could be going on.

Defo a bug.

[wintry spoke]

https://github.com/bevyengine/bevy/issues/15290

[summer talon]

Thanks! Really appreciate your testing this before the release.

[wintry spoke]

The way I'm using it, is that I collect all the clicks in the frame, then based on the selection mode, I filter out UI clicks, filter out anything that doesn't have a Selectable component, and then apply a heuristic that selects overlapping things based on how close the click was to that thing and then z top down, where if something is selected we select the next thing down in a stack. This makes clicking on a small bush at the base of a large tree more intuitive and also lets you cycle through several things (like resource nodes) that are stacked on top of each other.

I have a couple custom backends as well (to pick against the AABBs of spine skeletons) so I probably end up testing quite a lot of the new code just by using it.

I also use drag for box select and for selecting tile regions. That works, too.

[summer talon]

Do you happen to know if this is also happening with hover events Over & Out?

[wintry spoke]

Not sure. I can try to set up a test tomorrow.

[summer talon]

Ah, yep I think i've found the issue. Down & Clicks are actually both super broken rn, i think.

Do let me know tomorrow, in the meantime i'll write a test and get it sorted.

[wintry spoke]

If I'm in a drag and I cross over another entity, do you expect new drag start events to be emitted?

Something weird may be happening there, too, but I need to drill down more. I noticed box selection was restarting every time I dragged over a new pickable entity.

[summer talon]

that seems like a bug as well. I wouldn't expect that.

i think that may be related to the issue with Down and the state around pressing.

Currently there's a one-frame lag on Down events that's kind of throwing everything off, I think.

Here, let me put up a branch really quick and you can see if it's an improvement.

[wintry spoke]

Ok. might be a slight lag time in my ability to test because its evening and I have to start cooking dinner for the kiddos in a moment

[summer talon]

no problem, it'll be up for you to play with tomorrow if you'd like to.

[wintry spoke]

yeah i will

[summer talon]

https://github.com/NthTensor/bevy/tree/picking-bugfix-1 very quick first attempt, lets see if this fixes anything. there were three smallish bugs i think contributing to this issue.

state was getting cleared too early on pointer up (which explains why you only get the first event), and pointer down was using the previous hover map instead of the current one.

[knotty dune]

Q: Is there a migration guide yet?

[summer talon]

I made a very halfhearted attempt in the description for this pr.

If people are interested i may have some spare time to write a more complete version after the 25th.

The docs on on the bevy_picking module are now 95% complete and up to date as well.

[stoic flicker]

We still need to port / create some examples right?

And yeet Interaction

(and do a bevy_raycast equivalent)

[summer talon]

yes and yes (and yes)

i was looking into raycast a bit today, and decided i don't want to do it if i can avoid it

going to focus on the other two

tbh we can release without a mesh picker and tell people to use bevy_mod_raycast it will be trivial to port the existing backend.

i honestly don't know if we really need a non-colliders based mesh picker in tree

[stoic flicker]

I really want it for editor-y workflows

And for dumb jam games and beginners

[summer talon]

yeah but the editor can depend on ecosystem crates (or at least, i think we should let it) and we and stick it in the jam template?

I'd approve the pr but i wouldn't block the 0.15 release over it, and i would for a picking example and Interaction.

@thick umbra has a point that most engines only really expose users to ray-casts on colliders.

Although i guess they must also have non-accelerated mesh raycasts for their editors. 🤷‍♂️

[summer talon]

Draft PR up here.

[thick umbra]

Can we add mesh picking but leave general ray casting APIs out? I'd be fine with having the non-accelerated ray casts just for picking since it's not a too performance-heavy use case anyway

And doesn't have the risk of confusion with collider ray casting

[thick umbra]

I'm having trouble reading Godot's source, but I'm pretty sure they also just have accelerated physics picking (edit: I think they use separate editor gizmos, see later comment).

• I think viewport mouse input is handled here, pushing a physics picking event.
• Physics picking events for the viewport are handled in Viewport::_process_picking, which performs a point intersection test in 2D, and a ray cast in 3D. These queries use a BVH internally.
  If this is what it does, it must create colliders for the meshes somewhere (at least in edit mode) even if they're not actual physics objects. I haven't found where it does this though, so I'm not sure yet.

[thick umbra]

Unless this is only for runtime picking and only for physics objects, which is also possible. But I haven't found other picking logic

[thick umbra]

Nvm, Godot's editor seems to query a gizmo BVH for editor selections I think? No idea if meshes are also what Godot considers gizmos though (edit: looks like yes)
https://github.com/godotengine/godot/blob/694d3c2930bdfb43fd93e1e6641f66c4f19a5c77/editor/plugins/node_3d_editor_plugin.cpp#L742

Godot's source code is really hard to follow to me

[summer talon]

I would be fine with embedding a super paired down ray-caster directly into the mesh picking back-end. I suspect for the editor we will want to use gpu picking anyway.

But that requires gpu readback, which needs work.

[wintry spoke]

I'll test this in about an hour. Gotta get these kiddos to school first.

[summer talon]

Btw @stoic flicker is it ok if I mark some of the upstreaming tickets (like #12365) as complete?

it's really just that one i guess.

[stoic flicker]

Yep feel free

Spin off bite sized ones for the last tasks please?

[summer talon]

Blender's view-port apparently uses hardware occlusion queries btw, not cpu raycasting.

[wintry spoke]

This fixes the clicks. I get them all now.

Dragging is still weird, but I need to make sure that's not on my end. Looking deeper now.

[summer talon]

i'll take a look when i can as well, i had hoped it might get the drags but i'm not super surprised.

[wintry spoke]

When a drag ends, I get 4 drag ends (even though I'm only picking over one entity). When I am dragging and I drag over another entity, I get a new drag start.

Maybe this is intended. I think my expectation from other packages is that "when dragging, no more drag starts are emitted until a drag end and a new drag begins".

I'm not sure what explains the number of drag ends, actually.

For now, I can change my code to ignore drag starts if we're already dragging.

Is there a way to disable specific picking backends? I think I want all of my picking to use my bounds and aabb2d. Some sprites may have tighter picking bounds than the artwork, because we might want to ignore excess alpha. This also is just easier to manage, since every game entity that is pickable has bounds even though it may be rendered using different subsystems. I don't see how I can disable sprite_picking while keeping the rest of picking.

Maybe the various picking back ends could be features?

[stoic flicker]

Yes please. Can you open an issue?

[wintry spoke]

sure

https://github.com/bevyengine/bevy/issues/15306

[summer talon]

Def not intended. Will fix tonight after work.

[wintry spoke]

Another thing to look at. At the end of a drag, on drag end, a click is also emitted. A drag end should probably emit an Up, but not a Click.

In my case, this confuses my selection system because its like "great, drag is done, select all the things inside the box" and then immediately after "oh they also clicked the tilemap, deselect everything."

2024-09-19T15:53:26.826651Z  INFO tier1::selection::boxdrag: box selecting {3110v1#4294970406}
2024-09-19T15:53:26.826824Z  INFO tier1::selection::boxdrag: box selecting {3110v1#4294970406}
2024-09-19T15:53:26.826953Z  INFO tier1::selection::clicks: click!
2024-09-19T15:53:26.828228Z  INFO tier1::selection::single: process click
2024-09-19T15:53:26.828320Z  INFO tier1::selection: clear selection

the two "box selecting" are from duplicate "DragEnd", then the click! is the click being collected for that frame and by that time we're back in single selection mode which processes the click as a deselect

hehe

[summer talon]

oof, that's strange.

the drag state must be totally borked.

[knotty dune]

Got a problem: I decided to try migrating Quill from bevy_mod_picking, and quickly ran into trouble. There are a lot of places in Quill where an event handler triggers a one-shot system (widget callbacks in Quill are one-shot systems). Calling a one-shot requires a World, which can't be injected into an observer.

[summer talon]

Trigger it with a command.

I'm surprised to hear that. Do you need actual input/output?

[knotty dune]

It's fire and forget. Not sure what you mean.

[summer talon]

can you use Commands.run_system then?

[knotty dune]

OK I think that can work

Another issue: I relied on the behavior that On replaces itself when you re-insert the component.

[summer talon]

Ooh that's going to be tricky.

Let me think on that one. Totally solvable, but we probably need to improve to observer api to cover this case.

[knotty dune]

Here's the use case: you have an event handler which captures some state of the widget. When the state changes, you need to give it a new capture.

[summer talon]

I general I would try to model this sort of behavior with a single observer, and store the state as a component which the observer reads and modifies.

But I think we should probably have a way to support the current path.

The "replace the handler" paradigm works better with reactivity i imagine.

[knotty dune]

Here's an example:

Element::<MaterialNodeBundle<SwatchRectMaterial>>::new()
  .named("Swatch")
  .style((style_swatch, self.style.clone()))
  .insert_dyn(|m| m, material.clone())
  .effect(
      |cx, ent, (color, on_click)| {
          let mut e = cx.world_mut().entity_mut(ent);
          e.observe(
              move |_trigger: Trigger<Pointer<Click>>, mut commands: Commands| {
                  if let Some(on_click) = on_click {
                      commands.run_callback(on_click, color);
                  }
              },
          );
      },
      (color, self.on_click),
  )

The effect takes two arguments: a closure, and a set of dependencies. The closure gets run only when the dependencies change.

[summer talon]

I'll have to come back to this after my workday.

[knotty dune]

kk

[summer talon]

but thanks for posting the example, that's super helpful!

[knotty dune]

I suppose the effect could cancel the previous observer somehow.

The old mod_picking code worked differently:

.insert_dyn(
    |(color, on_click)| {
        On::<Pointer<Click>>::run(move |world: &mut World| {
            if let Some(on_click) = on_click {
                world.run_callback(on_click, color);
            }
        })
    },
    (color, self.on_click),
)

[summer talon]

Yeah. Need better apis for manipulating existing observers.

observers are just entities. You can create them manually with

world.spawn(Observer::new(|trigger: Trigger<Speak>| {}).watch_entity(my_entity));

you could try spawning them manually, tracking the entity you spawn on the watched entity, and then despawning/replacing the observer component on that entity.

in future I want better apis to get the observers watching a given entity

[knotty dune]

Yes, I've used that before. But replacing an observer requires keeping around additional state.

[summer talon]

Yeah. You'll need to link from the watched to the watcher, and replace the component on the watcher.

or possibly de-spawn/replace the watcher

it will be ugly

relations will make it better (i hope)

[knotty dune]

The problem with putting the state into a component is that it adds a lot of extra boilerplate and really degrades the ergonomics. In the example I gave, color and on_click are both parameters to the template - that is, they are being passed in by the caller. Having to copy them into a component means that the widget author needs to add code to both read and write an extra component.

[summer talon]

Yeah, i see the use-case. We need to make it easy to replace observers.

[knotty dune]

Something like e.unobserve::<Pointer<Cancel>>().

[summer talon]

yeah, we could 100% add a command for that.

or observe_unique which would drop all other matching observers at the same time that the new one is inserted.

i'll make a pr for both in the next week or so

e.look_away::<Pointer<Click>>()

[dusk spruce]

I would also like to see this, but what about use-cases where you don't want to end up over-writing observers inserted by the consumer? For example, a generic Button element that performs a fancy animation when Pointer<Click>'ed. If the user inserts their own observer for Pointer<Click> it can get overwritten by the element's logic. I guess you need to spawn the observer entity with a special marker for this case.

[knotty dune]

I'd be OK with a marker, either for the unique or unobserve case.

I'd use the same marker everywhere - that is, my widget library would define one marker which would be used on every observer in that library.

BTW, just as an aside, this is less of an issue in bevy_reactor than it is in Quill, because in bevy_reactor what the closure typically is capturing is a signal rather than a value, so the dereference happens inside the closure, which reduces the need for re-capturing. However, I suspect that re-capturing does occur in a few places (I'd have to look over the code to be sure).

[summer talon]

Alright, I've identified this issue too. I'm using output movements events in a place i should be using input movement events. Shouldn't be too too bad to fix, just a bit more tweaking and reorganization.

[summer talon]

Updated https://github.com/bevyengine/bevy/pull/15293 to hopefully fix this. haven't got time to build a test for it locally.

😅 sorry about that, in retrospect this issue is pretty obvious

[wintry spoke]

Doesn't build, but I may need something else?

// wants three in the tuple:
for (press_target, (location, _, hit)) in state.pressing.iter() {

// tuple has two:
pub struct PointerState {
    /// Stores the press location and start time for each button currently being pressed by the pointer.
    pub pressing: HashMap<Entity, (Location, Instant)>,

[summer talon]

it doesn't?? i just built it locally.

[wintry spoke]

Could be I'm missing something in my fork and need to reset it.

Let me look.

[summer talon]

you might just need to pull as well, i pushed some other stuff.

[wintry spoke]

got it working. with this i get 1 drag start and 1 drag end, as i expect. i still get a click when i end a drag over a pickable

[summer talon]

right, forgot about that one.

probably won't get to it tonight, have to pack for some travel

[wintry spoke]

May also want some kind of configurable drag threshold. Or might need to play with it. That click is probably covering when a drag happens on very small mouse deltas, like moving the mouse and clicking fast.

[summer talon]

The amount of drag shouldn’t matter, any OS press event followed by an OS move event should register. It’s odd that you are getting click events without a button up event.

Oh, it's happening at the end of a drag, right? I think I know what's going on.

[wintry spoke]

I'm not logging ups, they are probably happening. The question is when a drag ends, should it also emit a click.

[summer talon]

Seems like any motion between a press and release should many invalidate the click. Reading through mod_picking to see if I missed some logic for this.

It would be pretty easy to special case it so you never get both a click and a drag release.

still trying to understand the divergence frommod_picking. I'd rather this it by just properly preserving the original behavior.

I think this didn't happen before because Click always preceded DragEnd.

The fix might be to move it so the flow is Click -> DragDrop -> DragEnd -> DragLeave -> Up. picking wants to be naive, it fires the drag events but it doesn't actually know if the end user is treating stuff like a drag or not.

or Click -> Up -> DragDrop -> DragEnd -> DragLeave. @knotty dune & @stoic flicker do you have thoughts on the correct semantics for clicks here? If we can keep the current semantics with an acceptable ordering, i'd prefer that.

[knotty dune]

The only thing I care about is that drop comes before end. More generically, if some events are optional - for example, drop may or may not happen - then optional events come before non-optional events.

[summer talon]

And we agree it's correct to allow clicks be defined as [press -- move -- release] as long as it starts and ends on the same entity?

rather than forbidding moves

[knotty dune]

In the case of drag end/drop: typically, a drag end without a drop means "the mouse button released without dropping on anything", which is a signal to delete the gizmo and reset whatever internal state variables you were using. As an example, in my node graph editor, when you click on a terminal to drag a connection, it records which node and which terminal yoiu clicked on. When I get a drag end, then I reset these variables back to None. However, if I get a drop event, then I use those variables, along with the drop terminal, to create a new connection object which connects the two terminals.

[knotty dune]

I think so. I mean, you could implement Button using only drag/drop: a drag end with no drop means the action got canceled, whereas a drop means that the button was actually clicked.

[stoic flicker]

Yes, this will be really annoying otherwise

[summer talon]

you only get drag ends with movement, so a simple press-release won't trigger it.

[stoic flicker]

No strong preferences between these options

[summer talon]

cool @wintry spoke if the click happened before the drag ended, that would resolve this right? i am pretty sure this is why you didn't get this with mod_picking.

[knotty dune]

ok, so in that case you couldn't implement button with only drag/drop 🙂 Which is fine, I was just making a hypothetical.

[summer talon]

yeah i get what you mean. thanks both of you. good sanity check.

[wintry spoke]

yup. i don't process clicks in drag select mode when we're dragging

i mean, they get processed just nothing happens

[summer talon]

cool. this has made me realize there's a few cases were event ordering now differs from mod_picking in an unnecessary way. I'll get those switched around.

will probably do click-up and then drag events.

pushed.

[wintry spoke]

That fixed it. I don't see any issues now, with my current stuff.

[stoic flicker]

@wintry spoke wanna slap an approval on the PR?

I'll take a look too and we can get this merged

[summer talon]

Thanks again for testing this! The short feedback loop is really really helpful.

[wintry spoke]

Any time! I've got my game on bevy main right now, which I don't usually do so it makes it easy to test things.

[fluid sandal]

How are things going? I'm kind of coming up for air with the new baby. Don't really have much free time, but I might have some time to talk through any problems if that would be helpful. ☺️

[stoic flicker]

Generally quite smoothly!

[summer talon]

I’ll give you a summary when I’m back at my computer, but deterministic ordering has been implemented and now I’m writing some complex examples and showcases.

There’s some blockers we need to clear, and I want your opinion on allowing pointers to have a sequence of locations each frame.

But generally I think most people should be able to migrate now.

[knotty dune]

@summer talon A few more problems I'm having:

• I can't read signals in an observer, because signals require (non-mutable) World access.
• I get DragLeave events when I hover over a child of the entity I am dragging. I'm pretty sure this is not how it used to work in bevy_mod_picking.

For example, when I have a button, and I hover over the text entity inside the button, it thinks that it is outside. Disabling picking on the text is problematic because the text entity is passed in as a parameter from the caller.

[summer talon]

I’ll look into drag leave when I can

[knotty dune]

Would it be possible for observers to inject a DeferredWorld?

[valid pulsar]

They should already support taking a DeferredWorld as the only param

If not they definitely can

[knotty dune]

Ah, good, thanks

[knotty dune]

BTW @summer talon @valid pulsar - I've managed to integrate observers into Bevy reactor, things are working well but the code is kind of verbose and boilerplatey:

Element::<NodeBundle>.new()
    .observe(
        move |mut trigger: Trigger<Pointer<Click>>,
              mut q_state: Query<(&mut Pressed, Option<&mut Disabled>)>,
              mut commands: Commands| {
            let (pressed, disabled) = q_state.get_mut(trigger.entity()).unwrap();
            trigger.propagate(false);
            if pressed.0 && disabled.is_none() {
                if let Some(on_click) = on_click {
                    commands.run_callback(on_click, ());
                }
            }
        },
    )
    .observe(
        |mut trigger: Trigger<Pointer<Down>>,
         mut q_state: Query<(&mut Pressed, Option<&mut Disabled>)>| {
            trigger.propagate(false);
            let (mut pressed, disabled) = q_state.get_mut(trigger.entity()).unwrap();
            if disabled.is_none() {
                pressed.0 = true;
            }
        },
    )
    .observe(
        |mut trigger: Trigger<Pointer<Up>>,
         mut q_state: Query<(&mut Pressed, Option<&mut Disabled>)>| {
            trigger.propagate(false);
            let (mut pressed, disabled) = q_state.get_mut(trigger.entity()).unwrap();
            if disabled.is_none() {
                pressed.0 = false;
            }
        },
    )
    .observe(
        |mut trigger: Trigger<Pointer<DragEnd>>,
         mut q_state: Query<(&mut Pressed, Option<&mut Disabled>)>| {
            trigger.propagate(false);
            let (mut pressed, disabled) = q_state.get_mut(trigger.entity()).unwrap();
            if disabled.is_none() {
                pressed.0 = false;
            }
        },
    )
    .observe(
        |mut trigger: Trigger<Pointer<Cancel>>,
         mut q_state: Query<(&mut Pressed, Option<&mut Disabled>)>| {

A lot of the different kinds of events have similar injections, which have to be repeated for each event type.

[stoic flicker]

Nit: you can use Has<T> instead of Option<&T> if you're just checking if a component exists rather than looking at its value

@junior plank this is a good example of obscure but useful Bevy API

[junior plank]

I see, so you can observe a single entity to “latch onto it” with an observer and avoid looping through a large query

[stoic flicker]

I was talking about Has, but yes 🙂

[junior plank]

oooh i actually knew about that one! But thanks I learned something interesting about observer usage in practice instead!

[knotty dune]

One thing I wish observers had was a way to inject a component on the observed entity. Currently if I want to know if the button is disabled, I have to query all disabled widgets in the world and then pick out the one I want.

Also, didn't we discuss at one point folding the various pointer event types into a single combined event? I know that one of the reasons for doing this was event ordering, which got solved a different way, but that wasn't the only reason.

[shy sparrow]

There's a PR for better multi trigger support. Don't know if that's been merged.

[shy sparrow]

There's the idea to add a bundle filter on trigger but you can't read the components values with that. I would prefer this: #ecs-dev message

[stone bone]

#14674?

[shy sparrow]

that's the one

[glass oasis]

Hello ; I'm studying updating bevy_mod_picking to bevy main (and 0.15) ; would that still make sense or the API you're cooking here has been particularly studied to replace bevy_mod_picking ?

From my understanding it was considered to have both solutions somewhat compatible, but I'm not sure.

To help the ecosystem I thought migrating bevy_mod_picking with a focus on ease of migration would make sense to ease migration from crates dependent on it, even willing to someday switch to bevy's now built-in picking 🤔

[summer talon]

My goal would be to have people port over to this, it’s basically just mod_picking but using observers. Some people have already done this. It’s kind of up to @fluid sandal what to do now. Currently we’re not sure if we can/should upstream stuff like the mesh-picking backend.

[turbid arch]

I followed a rabbit hole: I wanted to see how far away we are from having a great editor cam like https://github.com/aevyrie/bevy_editor_cam/tree/main in Bevy. Now that we have the editor prototype repo I thought I'd try to put bevy_editor_cam (aevyrie) into bevy_editor_camera (bevy_editor_prototypes).
(Also I've asked about upstreaming before and they were OK with it).

I swapped out bevy_picking_core and used bevy_picking and added a couple of small missing things in bevy (likely not ported yet, I haven't followed this working group closely).

Eventually (after a few crimes) I got the cad example running without crashing (for some reason it is shaking). But the cam doesn't work, so I did a bit of digging.
Now I know we don't actually have backends for this: We only have sprite and UI (?).

Is anyone working on a raycast backend?
I saw aevyrie's raycast repo has ~300 commits so I think it's probably not a trivial thing to do, but I don't know how far along the current Bevy Ray2d/Ray3d APIs go either?

[humble fog]

btw, we've been shipping this camera controller in our stuff at foresight and it's absolutely amazing to use in practice.

[turbid arch]

You mean bevy_editor_cam? I use it for everything too, I love it

[humble fog]

yep

[stoic flicker]

Oh right, I forgot this was permissively licensed. Adding an issue for the editor prototype

[thick umbra]

Is anyone working on a raycast backend?
Not yet, as far as I know. The plan here is to add a mesh picking backend, but not expose general ray casting APIs to users yet, since they (imo) would be much more controversial and require design w.r.t. how things interact with physics colliders and how the queries are accelerated

[stoic flicker]

Yep, this is a good plan for now

[turbid arch]

Does mesh picking mean AABBs only or something more?

[thick umbra]

The initial version would likely just par_iter over visible meshes, test for ray intersections against their AABBs, and then test for triangle intersections for the meshes whose AABB was hit

And return relevant hit data

[turbid arch]

Is there fallback behavior for misses? Thinking of the CAD example above if you don't try to anchor the camera on the object itself but the backdrop (which is at an "infinite" distance)

[humble fog]

IIRC bevy_editor_cam uses the last hit distance instead of assuming infinity

[turbid arch]

Ah clever. In that case the mesh backend is likely enough for the CAD example I'd think

[fluid sandal]

I saw aevyrie's raycast repo has ~300 commits
I wouldn't worry too much about that, most of that is 1) it's been around through many bevy updates, it's an old crate 2) it's had a lot of API improvements, and 3) I've done a decent amount of optimization work using bevy-only built-ins, no special acceleration structures or physics libs.

The crate is pretty tiny, especially if you only keep the immediate mode raycast systemparam, I think it would be a bit silly to reinvent the wheel when that crate exists, and has already seen a lot of real world use.

Though, tbh, we should really be focusing effort on getting the shader picking backend merged. It's been revived and died multiple times.

[fluid sandal]

Re: (2) #general message

@aevyrie I'm playing around with your bevy_mod_raycast crate and just wanted to say thanks, and how much I'm enjoying it. I played with it a bit back in January 2023 and it was really helpful then, but using it now I'm shocked how much it has improved in that time. Cursor raycasting is so much simpler, really great DevEx. Awesome stuff 🙌

[turbid arch]

I think it would be a bit silly to reinvent the wheel when that crate exists, and has already seen a lot of real world use.
Yep it would be great to have ported. Would need consensus though.

getting the shader picking backend merged. It's been revived and died multiple times.
I made the first one. RIP.

[stoic flicker]

I'm on board with porting bevy_mod_raycast as is and then replacing it with bevy_math APIs gradually

[fluid sandal]

Shader picking is still preferrable IMO, and not controversial at all. The work has pretty much all been done.

[humble fog]

I'm not sure shader picking should be the default. There's a lot of pitfalls with it and it can easily break with custom materials if users don't support it. It should be an option, but not the only option.

It also doesn't work for transparent meshes without OIT to sort it correctly

[fluid sandal]

Yup. It seems like the "correct" way to do things from what I've seen though. That, or using your physics engine for raycasts.

[turbid arch]

Flashbacks to debugging GPU picking in renderdoc when I found picking IDs were being blended

[humble fog]

Uh, actually, not sure about that, it might work without OIT

[fluid sandal]

Shader picking should scale much better than raycasting, and doesn't require you have a physics engine. The cost is latency.

[thick umbra]

The crate is pretty tiny, especially if you only keep the immediate mode raycast systemparam, I think it would be a bit silly to reinvent the wheel when that crate exists, and has already seen a lot of real world use.
Yeah my idea was that we'd just upstream the core ray casting logic from bevy_mod_raycast for the picking backend, but leave the system param and deferred API out. Or, if we really wanted the system param, at least call it something like MeshRaycast.

As long as it's 100% clear it's not for colliders since all other engines just expose physics raycasts

[stoic flicker]

MeshRaycast is very clear

[fluid sandal]

We can paint that bikeshed whatever color we want. 😛
Though, the API isn't tied to meshes, it could be doing raycasts using a physics enginer or whatever else under the hood if you wanted.

I don't really have a horse in this race though. My only opinions are (1) shader picking is probably what bevy should ship by default and support with its built in materials and (2) if you are going to add mesh raycasting, you might as well steal from mod_raycast.

That's the beauty of picking backends though, you can use whatever combination makes sense for your use case, and I can maintiain custom ones out of tree for work.

[summer talon]

That seems like the perfect approach

[humble fog]

It also increases the memory bandwidth use because you're sending a u64 for each entity being rendered and I'm not sure if all targets support having multiple render targets which is needed when doing shader picking. But tbf, it's not like a cpu based raycasting backend doesn't have perf impacts either.

[fluid sandal]

we could probably get away with a u32. I don't think you would ever have two entities with the same id and different generation rendering on screen at the same time (edit, the issue is actually latency, once you are on the CPU side, you need to be able to look up the picked entity on the GPU, which might be a frame or two delayed)

Or you could hash it down to a single u32, and look it up when you are back CPU side

Collisions there seem extremely unlikely.

[humble fog]

That's essentialy what I did in my PR, but there's a cost to doing that too. I think, whenever I or someone else picks up the PR again we should go with the raw u64 entity id first. Then implement any kind of compression scheme in a separate PR with benchmark to compare it

[turbid arch]

I didn't follow the adopted PR, what's the TLDR of why it stalled (or is it closed)?

[humble fog]

Lack of time/energy/motivation to finish it at the time and now it's been so long that it would probably be easier to rewrite it again

It touches a lot of parts of the renderer that have changed a lot since then

All the mesh preprocessing stuff is very differnet now

[fluid sandal]

looks like there is existing precedent for maintaining a lookup table

Each object in each layer gets its own picking color assigned. The picking color is determined using layer.encodePickingColor() that converts the index of a object of a given layer into a 3-byte color array (the color buffer allows us to distinguish between 16M unique colors per layer, and between 256 different layers).

After the picking buffer is rendered, deck.gl looks at the color of the pixel under the pointer, and decodes it back to the index number using layer.decodePickingColor().
https://deck.gl/docs/developer-guide/custom-layers/picking

[turbid arch]

I see they do this: https://github.com/visgl/deck.gl/blob/9067f45a7d9424e397bc823ad89f3e010ddf1b31/modules/core/src/lib/layer.ts#L404
Which rang a bell, looks close to what I was doing too: https://github.com/bevyengine/bevy/pull/6991/files#diff-95752be9c726e25380ae41f49409451b3a7a9a4ce8878f84dc0c896646b9c343R16

[turbid arch]

I took a look at bevy_mod_raycast, you mention leaving immediate out. From the project structure it's obvious what you mean there since there is a module called that.
But you said "leave the system param.. out", do you mean this? https://github.com/aevyrie/bevy_mod_raycast/blob/main/src/immediate.rs#L143

[fluid sandal]

If I was porting it, I would only port the immediate mode thing with the Raycast system param.

It doesn't require any systems or plugins to work.

[thick umbra]

I meant that we should leave the system param out of the public API, at least under its current name. It can exist internally, or be exposed under an explicit MeshRaycast name

I agree it makes sense to only port the immediate mode version though

[turbid arch]

Imo it seems that if it's clear what to extract from bevy_mod_raycast then a simple mesh picking backend seems close, which then enables bevy_editor_cam to be upstreamed without too much work

The largest stumbling block of having bevy_editor_cam compile against main was that OrthographicProjection has changed and doesn't have a scale field anymore.
Not sure if you've looked at that @fluid sandal ?

[fluid sandal]

oh no

what happened to scale?

@turbid arch

[thick umbra]

https://github.com/bevyengine/bevy/pull/15075

scaling_mode is what you're intended to use

[fluid sandal]

welp, fingers crossed this doesn't break a ton of stuff at work

[turbid arch]

It wasn't obvious for me what to do since bevy_editor_cam has ortho.scale in a lot of places.
And ScalingMode (enum) has a lot of variants

[fluid sandal]

oh god, this is going to be a huge headache

[stoic flicker]

We can revert that PR if you'd like: I just wasn't keen on having three distinct ways to scale orthographic cameras

[turbid arch]

Hm is maybe this enough? https://github.com/bevyengine/bevy/pull/11030

[fluid sandal]

I don't really know enough tbh. It looks like the right choice, it's just gonna cause a lot of churn for me personally.

If anything, it will probably result in fixing some bugs caused by all the scaling properties that I didn't know about.

[turbid arch]

Ah nvm the output of that mul is still scalingmode

[fluid sandal]

I'm just anxious because all the ortho code has been a big PITA to make it work seamlessly with perspective.

[glass oasis]

I could make it all compile and zoom still works ; I'm wildly unfamiliar with all that code (and on a trackpad currently) so there's most definitely wrong shortcuts I took, but I'm hopeful 🤞 (https://github.com/aevyrie/bevy_editor_cam/pull/19) /offtopic

[knotty dune]

Idea: on the web, when an event bubbles up the DOM hierarchy, if it makes it all the way up to the top and falls off the end, the event is then dispatched to the window object. This is often used for global "catch-all" handlers. I'm wondering, could Bevy do something similar? Of course, there could be more than one window, but I'm assuming that the events would be dispatched to the primary window.

The use case is actually more compelling in the case of Bevy because, unlike the web, there isn't a single document root; if you have multiple UIs, you'd need to install a catch-all handler on each one. It would be easier if we could just install an observer on the window entity.

[main rune]

could be a catch-all per window and untargeted event for catch-all all windows

catching events from different windows in the primary window would be a bit weird

[old venture]

I don't know exactly how event propagation currently works with mod_picking, does propagation stop once a handler is found?

On web you must call stopPropagation() to stop it from your handler.

I'd be curious what the performance impact of "default propagates even if handled" is on deeper UI hierarchies. I assume it's an exclusive system so it stops multi threaded execution.

[knotty dune]

• bevy_mod_picking has stop_propagation().
• observers have trigger.propagate(false).

Although the docs don't really say what happens when an event bubbles all the way up. Does it call the global observer?

[fluid sandal]

https://github.com/aevyrie/bevy_eventlistener/tree/main?tab=readme-ov-file#performance

[stone bone]

The observers will propagate through Parent until either they are stopped or there are no more parents

[knotty dune]

Yeah, what I'm thinking now is - forget windows. Just have a way to define a catch-all observer that can receive any bubbled events that fell off the end of the chain, that is, never got propagate(false) called on them.

BTW the only reason I'm discussing this here is because this where event bubbling has mostly been discussed. However, my use case isn't about picking, but about keyboard event bubbling.

[stone bone]

That would be fairly easy to do either by modifying &Parents Traversal impl or adding a new traversal impl because we don't necessarily always want that behaviour

[wintry spoke]

This is a churn and make work heavy update. Just moving from add to queue caused me to touch 576 lines of code. Moving from &mut self commands to self was another bucket load. The scale stuff breaks the camera and inverts the math, etc. got it all working but you'll have to grab coffee and grind it out. So game dev.

I couldn't search and replace .add to .queue because of meshes.add, etc hehe

[summer talon]

I'm still on vacation, but this is actually the first thing I implemented as part of my picking examples work.

Will publish as part of a few prs when I come back out of the jungle. Just adds the window entity as a hit behind everything else.

/// Generates pointer hit events for window entities.
///
/// A pointer is treated as hitting a window when it is located on that window. The order
/// of the hit event is negative infinity, meaning it should appear behind all other entities.
///
/// The depth of the hit will be listed as zero.
pub fn update_window_hits(
    pointers: Query<(&PointerId, &PointerLocation)>,
    mut output_events: EventWriter<PointerHits>,
) {
    for (pointer_id, pointer_location) in pointers.iter() {
        if let Some(Location {
            target: NormalizedRenderTarget::Window(window_ref),
            ..
        }) = pointer_location.location
        {
            let entity = window_ref.entity();
            let hit_data = HitData::new(entity, 0.0, None, None);
            output_events.send(PointerHits::new(
                *pointer_id,
                vec![(entity, hit_data)],
                f32::NEG_INFINITY,
            ));
        }
    }
}

tide you over until then

[wintry spoke]

Congrats on completing the core PR for this project.

[thick umbra]

Did anyone end up working on the mesh picking backend yet? If not, and we would like it to make it in for 0.15, I could probably give it a try over the weekend

-# no promises tho

asking because of #engine-dev message and the looming RC

[stoic flicker]

Not yet, and strong yes please

I'll take the Interaction-yeet PR

and then that's all the required features

Examples would be really nice to have, but we can piece those together throughout the RC

[summer talon]

I tweaked the traversal trait so that we can have events fired on parentless entities propagate to the window.

behold by beautiful cursed child

#[derive(QueryData)]
pub struct ParentThenWindow {
    parent: Option<&'static Parent>,
    window: Option<&'static Window>,
}

impl<E> Traversal<Pointer<E>> for ParentThenWindow
where
    E: Debug + Clone + Reflect,
{
    fn traverse(item: Self::Item<'_>, pointer: &Pointer<E>) -> Option<Entity> {
        let ParentThenWindowItem { parent, window } = item;

        // Propagate event to parent, if it has one.
        if let Some(parent) = parent {
            return Some(parent.get());
        };

        // Otherwise, propagate to the window entity (unless this is a window entity).
        if window.is_none() {
            if let NormalizedRenderTarget::Window(window_ref) = pointer.pointer_location.target {
                return Some(window_ref.entity());
            }
        }

        return None;
    }
}

this will also let you do some very cursed stuff, controlling where stuff propagates based on the event data (which is mutable by observers). fun stuff.

this plus the window picking backend should get us pretty close to the web behavior. Writing examples, it turns out it's super useful to be able to register a handler that says "give me all clicks on this window, even if it's already been handled" for de-select logic.

(this is better than a global observer, because you can stop the propagation, a global observer will always get hit first, this is conditional and after other handlers run)

[knotty dune]

All right, that will be useful. I plan to use this for things like tab-key focus navigation.

[summer talon]

i needed it to cancel a pop-up menu when you click away.

thanks for pointing out the need for this

[knotty dune]

I've implemented a framework for keyboard event propagation based on my earlier discussion: https://github.com/viridia/bevy_reactor/blob/main/crates/bevy_reactor_obsidian/src/input_dispatch.rs

#[derive(Clone, Debug, Component)]
pub struct FocusKeyboardInput(pub KeyboardInput);

impl Event for FocusKeyboardInput {
    type Traversal = Parent;

    const AUTO_PROPAGATE: bool = true;
}

Keyboard events get triggered on the current focus element (or if there is no focus, it will be the window).

[summer talon]

sweet

[knotty dune]

With windows as handlers-of-last-resort, you just need to install a keyboard observer on the window - and you can have different handlers for desktop-style or console-style focus navigation.

Demo of keyboard navigation and toggling

[summer talon]

btw, have you thought about

.observe(|mut trigger: Trigger<Pointer<Down>>, mut commands: Commands| commands.run_system_cached_with(disable_self, trigger))
.observe(|mut trigger: Trigger<Pointer<Up>>, mut commands: Commands| commands.run_system_cached_with(disable_self, trigger))
...

[summer talon]

gorgeous

[knotty dune]

Hmmm, worth thinking about

[summer talon]

i'm basically building components out of cached oneshots

[stoic flicker]

Do we want parent -> window, or parent -> global?

[summer talon]

global observers always run even when triggered on specific entities

[stoic flicker]

Ah, kk

[knotty dune]

We would need to invent some new kind of global handler

[stoic flicker]

Yeah

[summer talon]

and they run before the entity observers do, iirc

which is sus

[stoic flicker]

We can just do windows as root then 🙂

[summer talon]

we may want to have a concept of "interaction zones" which replaces this eventually, with the default interaction zone being the window

this is how ios works

kinda

[knotty dune]

This is good enough for now. There are some advantages to a special "last chance" global handler: it means that libraries who want to be last on the food chain don't need to know about how many windows your app has.

With a global handler, a library can just export a Plugin that says "give me these events". With a window, you need to register the last-chance handler on each window. But even that is worlds better than having to install the last-chance handler on the root of each pickable entity tree.

Here's a link to the discussion which lays out the strategy for keyboard focus handling: https://github.com/bevyengine/bevy/discussions/15374 .

Implementing this is not very much code, less than 100 lines I'd guess.

[summer talon]

if you're cool with it, i'd like to steal it (put it in picking probably behind a plugin setting)

[knotty dune]

sure

another thing we need to sort out is wheel events and trackpad scrolling - I want these to bubble as well, so that it scrolls the region you are hovering over.

[summer talon]

that's probably going to need to be very opinionated and abstract, depending on the degree to which we want touch to play nice with it.

[knotty dune]

Like if you have an editor with a preview window and an orbit camera, you probably want the trackpad scroll to rotate the camera when the mouse is over the 3d preview, and not have it move the camera when the mouse is over a scrolling list of items.

You basically send "wheel" events to the current hover entity, and observers can decide to intercept it or not

[summer talon]

are you dispatching Focus/Unfocus observers? it doesn't look like you are, i guess you are doing reactive stuff from the focus resource?

[knotty dune]

That is exactly right

let focused = builder.create_focus_visible_signal(button_id);

[summer talon]

ah yeah, interesting. I'm playing around with fully observer-based focus stuff. we'll have to compare notes once my system is a bit more developed.

[knotty dune]

This is a Signal<bool> which is true when button_id is focused.

[summer talon]

fancy ✨

[knotty dune]

I have been busy re-writing bevy_reactor to incorporate ghost nodes, the whole design has changed.

[summer talon]

i have not been able to read any of that since i am on vacation but it does sound very appropriate for 👻 spookey season 🎃

[knotty dune]

Haha

[thick umbra]

Where should a mesh picking backend live? Putting it in bevy_render doesn't necessarily make sense other than putting it close to Mesh, but it also seems to be a circular dependency so it wouldn't work anyway

[stoic flicker]

Let's do it's own crate for now

[thick umbra]

What should it be called? bevy_raycast would be misleading imo

bevy_mesh_picking?

[stoic flicker]

Exactly

[knotty dune]

Or bevy_picking_mesh so that it sorts next to bevy_picking.

[stoic flicker]

I've written up the remaining work for 0.15 for this group into issues and added them to the milestone: https://github.com/bevyengine/bevy/issues?q=is%3Aopen+is%3Aissue+label%3AA-Picking+milestone%3A0.15

I'll take the Interaction refactor and tank the X-Controversial 😅

[summer talon]

I will open my window-picking changes tomorrow. I think it should be on the milestone too when it goes up but I guess it might be a bit late.

[stoic flicker]

Nah feel free 🙂

I think we should be able to get that in

[fair raptor]

Did we all the ordering bugs get fixed already?

I saw a bunch of different people reported issues with ordering or missing events when testing the crate

[stoic flicker]

I think so, nth made a big PR to resolve it

[knotty dune]

@stoic flicker Some notes on how I do button press (my replacement for Interaction):

First, I treat hover and "pressed" separately, rather than having a single component that represents both states. For buttons, I want to correctly handle the "roll-off" behavior - that is, if you press a button, but move the pointer outside the bounds of the button, the click event is canceled. For widgets that toggle, like checkboxes, toggle switches, disclosure toggles and so on, this is less important because if you accidentally click you can just click again to revert the action.

For hovering, I don't use the picking events, but query the hover map directly. This has two advantages: first, it means that I don't accidentally "miss" an enter or exit event if, for some reason, it gets intercepted by some child entity. Second, for most buttons, the logic we want is "hover within", meaning that we want to know whether the mouse is hovering over the button or any of it's descendants. We don't want to have to track enter and leave events for all descendants.

I have a friendly API that hides all of this logic: you simply call builder.create_hover_signal(entity_id), this returns a reactive Signal<bool> which is true whenever the widget is being hovered. There is a corresponding signal for keyboard focus as well.

In fact, for buttons there are actually four boolean flags that represent the button's state, not just two:

• hovered
• pressed
• focused
• disabled
  All of these affect the button's appearance - for example, the button appears grayed out when disabled. Exactly what visual effects will be enabled by these states will be different for different games and/or widget libraries.

For the pressed state, I previously used a Mutable, but I recently changed it to insert a Pressed marker component on the button.

The reason for this is because I can think of use cases where an external system might want to query buttons that are pressed. I don't do this for hover, because I consider that to be an internal feature of the button, and not exposed to the outside world.

The Pressed component is inserted and removed by the various picking events. Note that we have to handle DragEnd, because we want to handle correctly the case where a pointer up event happens when the mouse is outside the button bounds - normally pointer up events would not be sent to the button in that case.

The actual button action is performed on the picking Click event, which duplicates some of the logic I have described - that is, cancelling the click in the roll-off case. The reason for the duplicate logic is that I'm using the picking events to update the button appearance: that is, I want the button's visuals to accurately telegraph to the user what is going to happen when they press or release the mouse button, and that will change based on the current mouse position.

For buttons, ENTER and SPACE are equivalent to click, if the button currently has focus.

Note that, according to WAI-ARIA guidelines, disabled buttons can have keyboard focus, because otherwise screenreader users would have no way to discover that these buttons exist. However, clicking (or ENTER/SPACE) on a disabled button does nothing, no action is performed.

Bottom line: people think of buttons as the simplest possible widget, but they are surprisingly complex!

[summer talon]

Theoretically, yes. There may be bugs with the new implementation still but those have mostly been very easy to fix.

[stoic flicker]

@knotty dune @fluid sandal I've spent most of my day chewing on the Interaction-yeet design: https://github.com/bevyengine/bevy/issues/15550#issuecomment-2387057725

I think that a variant on DreamerTalin's approach above is likely best, but I want to get my hands dirty before locking that decision in

@knotty dune can you send me your code to work off of?

Implementation for that is my primary work tomorrow

[knotty dune]

https://github.com/viridia/bevy_reactor/blob/main/crates/bevy_reactor_obsidian/src/controls/button.rs

You'll want to look at the various observe() calls.

[knotty dune]

Some random comments on your doc:

• Because of the naming collision with the Disabled marker comp, one possibility might be InteractionDisabled (a bit wordy perhaps).
• I think that drag and drop is / should be solved separately. "drag sources" and "drop targets" have their own set of behaviors.
  • In general, you have "self-dragging" widgets like sliders, which handle drag operations but don't drop on anything; and you have "DnD" widgets like dragging rows in a list to change the order. In JavaScript these are actually handled with two different kinds of events (pointer-capture vs. DnD events).

• In addition to tooltips, you can potentially have things like a footer / status bar which gives a description of the currently hovered item. In other words, the concept of what a "tooltip" is is much broader in a novel game-like UI, and doesn't necessarily mean a floating bubble.

[stoic flicker]

Because of the naming collision with the Disabled marker comp, one possibility might be InteractionDisabled (a bit wordy perhaps).
Yeah, it can't be Disabled but it needs a good name

Helpful comments though

[knotty dune]

Note that a good widget library will also set the a11y disabled property, so that the screen reader will read "save file button disabled"

(I have not done this yet)

[stoic flicker]

I wanted to make sure I wasn't missing any critical use cases. It really seems like the dynamic styling is really the only one where we want the "is this button currently hovered" data for every entity, but that's an important use case that we can't neglect

(and it's like the only thing that Bevy's existing design is any good at doing lol)

[knotty dune]

A lot of modern widget libraries on the web don't even show effects on hover - as part of the trend towards "minimalism" and "mobile-friendly", hover effects are often left out.

Which is not to say that UI designers aren't animation-crazy, but they spend most of their time thinking about transitional animations - being able to fluidly express the change from one application mode to another with things like screen wipes.

[stoic flicker]

In games, I generally see hover + focus getting smooshed into one effect, because of how important it is to show focus for gamepad controls

I also think tooltips are wildly important for both games and tooling

[knotty dune]

For obsidian, I use outline for focus, it's easy and cheap.

[summer talon]

I think Interaction and in general components should implement their own state tracking.

[summer talon]

i thought you were talking about the note-taking program for a second and was confused

[stoic flicker]

Right, so we move the data onto Button?

[knotty dune]

"obsidian" is the name of my opinionated widget library - it's intended for building editors, it uses the color palette from the Bevy editor mocks, it looks a lot like egui.

[summer talon]

that would be the first thing i try, yeah.

I wish there was an easier way to add an observer for an entire 'class' of entities.

like, a single observer that targets anything with the Button component.

it's possible to do manually with hooks but awkward

To get the best performance, we should really only spawn one observer for each function and just manipulate it's targets.

[knotty dune]

Alice, another thing you need to decide is which events stop propagation. For my widgets, I always stop propagation for any event I handle, so that parents don't accidentally handle an event which has already been handled by a child. However, this is controversial with the Click event, because there are two ways of handling clicks.

• One way to handle clicks is to let it propagate, and the parent handles the click event to decide which button was clicked.

• However, I don't do that - instead, the button widget is passed an on_click handler, which is a one-shot system

This second method is more flexible, in that it allows entities other than direct parents to be notified of the click event. But it also requires additional wiring.

I've had bugs in the past (in web apps) where I forgot to stop propagation on a child element, and it was handled by the parent.

Take the case of keyboard focus: every focusable widget, on mouse down, will set focus to itself. But how do you clear the focus? By clicking on the backdrop. This means that the backdrop / window / top-level also needs a mousedown handler which clears focus. But you don't want this to happen if you clicked on a button - so the button needs to stop propagation of the mousedown.

[summer talon]

i think we probably want to stop propagation when handled. You can use picking "transparency" to allow the same click to propagate up multiple time, or multiple trees.

but that's something to test in the rc

[knotty dune]

My focus system is divided into two modules: one for basic operations on focus and for dispatching key events to the current focus element, which is here: https://github.com/viridia/bevy_reactor/blob/main/crates/bevy_reactor_obsidian/src/input_dispatch.rs - the FocusKeyboardInput is a wrapper around KeyboardInput which enables bubbling.

[summer talon]

oh, and even if you stop propagation all the other Trigger<Pointer<Click>> observers on that entity always run.

so you don't need to pass callbacks.

[knotty dune]

The other module implements tab-navigation, which is a separate module because there can potentially be different strategies for focus navigation depending on platform.

We've discussed upstreaming the basic keyboard focus stuff but there's some unresolved issues - like where this should live, what dependencies it can have (it needs bevy_a11y and bevy_hierarchy, neither of which bevy_input currently has.)

[summer talon]

i think it should probably live in it's own bevy_focus crate.

[knotty dune]

Similarly, I also want bubbling for MouseWheel events, so that trackpad scrolling can be routed to whichever entity the mouse is over - whether that be a scrolling list, or a 3d preview window with camera orbit controls.

[stoic flicker]

Yeah, this is my feeling too

Then have bevy_a11y consume it

[knotty dune]

There's also AutoFocus, which is a marker that indicates that a widget wants to be focused on spawn. This is used for things like the default button in dialogs.

[stoic flicker]

@summer talon do we have a unified pointer event type on main yet?

I'm not seeing one in https://dev-docs.bevyengine.org/bevy_picking/events/index.html

[summer talon]

erm, no. we did observers to avoid needing that.

[stoic flicker]

great, I can work with that 🙂

[summer talon]

we can't send both unified and non-unified events at the same time (because stopping propagation on one wont stop the other)

so it has to be a breaking switch to an enum, not addative.

what'cha working on?

[stoic flicker]

The Interaction-yeet PR 🙂

Trying to get button state set via picking events

[summer talon]

my recommendation would be to use multiple observers currently. it's not ideal, but larger changes to the observer api didn't seem in scope for this group.

i'd rather fix it on the observer side than the picking dispatch side.

[stoic flicker]

Yep, that's my plan!

pub fn button_hover_observer(
    trigger: Trigger<Pointer<Over>>,
    mut button_query: Query<&mut Button>,
) {
    if let Ok(mut button) = button_query.get_mut(trigger.entity()) {
        button.hovered = true;
    }
}

[summer talon]

yeperino

[stoic flicker]

I have a lot I would like to improve about our buttons

But I'm going to try and restrain myself to the simplest set of changes possible in this PR

(I really want to store the on-press callback on the widget and use observers for that)

[main rune]

but then it would collide with any other callbacks you add would it not

[stoic flicker]

No, as a field in Button

Might actually just use one-shot systems

But, another PR

[summer talon]

i'm not convinced storing a system id for a callback is better than letting people register additional Click observers.

[main rune]

Can't wait for people to query every existing button and add on_click observer on each one

[summer talon]

talin is only doing it because it makes it easier to swap out the handler re-actively

we can fix that through the observers api, IMO.

[stoic flicker]

Alright, interaction-yeet is ready for the first round of reviews, checking that I got the basic strategy and example migration right

@summer talon, I'm running into problems where text children are not bubbling up the pointer events correctly. How do I fix that?

[summer talon]

you are? can you link me to some sample code?

are text entities not in the normal hierarchy?

[stoic flicker]

Yeah. Try the button example

[summer talon]

are you sure they are actually not bubbling, and it's not an issue with the ui picking backend?

[stoic flicker]

Oh wait, maybe it's just that example being cursed

Argh, yep...

[summer talon]

the bubbling code is extremely simple, if it's not working then either the hierarchy is broken or there's UB or I have no idea what

[stoic flicker]

Yeah, this example is just kinda dumb I think? Cleaning it up

Oh that's fun: if your cursor leaves the window and then you release it your button gets stuck pressed

[summer talon]

yes, known issue.

this will have to be resolved upstream, I think, since we don't get events when the cursor leaves currently i think.

when the winit pointer api lands, I think moving the cursor off the window will produce a cancel event, just like the os canceling a touch gesture.

[stoic flicker]

Yeah. Distinct issue IMO

I should make sure that button press -> move off the button -> button release releases the button though

[summer talon]

what's our desired roll-off behavior?

[stoic flicker]

Release IMO

And then handle most click behavior on mouse-down

[summer talon]

then we need to react to Click and cancel focus on Out unless we first receive a Down and a DragStart in which case we need to wait for DragEnd.

i can check this over.

[stoic flicker]

kk

I thought covering drag behavior was out of scope for the simple buttons though?

[summer talon]

click should be click, focus and press state needs to be sticky on drag basically, right?

[stoic flicker]

Okay, so it doesn't seem to be the example's fault. It's happening for all of our examples.

Pointer presses are getting propagated correctly, but pointer hovering seems to be blocked by Text children

[summer talon]

ok answer me this: user presses pointer on button, user moves cursor off button, user releases pointer.

when does state change and what happens?

i was thinking state would change on release, because if they move back on to button we still want to register as a click

so press state needs to remain on press->move-off, and internally this is modeled by a drag

[stoic flicker]

State changes as soon as the pointer is pressed, state resets as soon as the cursor move off, state doesn't change when they release

If you also need to handle drags, we care about clicks

[summer talon]

are we sure we wait to reset on move off not on release?

[stoic flicker]

I think that's fine for widgets where we don't care about dragging

[summer talon]

just played around with a html button, your right that's how it works.

yeah, nevermind about dragging

i had the state machine mixed up in my head

[stoic flicker]

I'm still blocked on the Pointer<Over> propagation being weird BTW

And don't know where to look

[summer talon]

let me take a look

interestingly, for web buttons if you pointer-press and only then move over the button, it enters the pressed state.

[stoic flicker]

We don't replicate this but I don't think we care to

[summer talon]

oh 🙃

so what we're seeing isn't just that stuff isn't bubling. It's that Out is firing when we mouse over the text.

[stoic flicker]

Oh geez

[summer talon]

when we move to a component to it's child, we fire Over for the child (which bubbles up and does nothing) then Out for the parent, which wins.

there's a few things that could be going on here.

i think it's just that text shouldn't be being picked here.

adding Pickable::IGNORE to the text should fix it

[stoic flicker]

Okay, lemme try that

It works!

[summer talon]

that's a rough edge we can sand off when we start to make more components

[stoic flicker]

Yeah 🙂

[summer talon]

fixing ordering for Over/Out will also resolve this

kinda maybe sorta

[stoic flicker]

That would give us spurious change detection yeah?

[summer talon]

correct

btw press->mouse over does result in the hover state, so we are actually compliant with web here.

I think the right way to turn this into a component would be to have a ButtonClicked event which you add an observer for on the entity, which is actually triggered by Up.

my main concern with IGNORE is if we later make a text-selection thing backed by picking, things will get complex. But button text should never be select-able anyway so down the river that issue goes.

[knotty dune]

That's not the primary reason. The primary reason is because the ultimate handler of the event:

• Might not be a parent
• Might not know it's coming from a button
• Might not know it's a click event.
  My philosophy of event handling is that a ui hierarchy is also an abstraction hierarchy: widgets transform low-level, hardware-specific events ("click") into high-level, application events ("confirm save file").

[summer talon]

ah, reasonable. i think we can still achieve those goals with observers

[knotty dune]

I don't like this solution, because buttons won't have just text, it might have icons too. A generic "button" template will take children as a parameter - to require that all children passed to a button (or checkbox, or any other widget that has a "children" parameter) also be Pickable::IGNORE is an onerous constraint on users.

[summer talon]

yeah it's not great. i'm not sure what the best way to attack this is. we could add a component that blocks bubbling from bellow, or we could add a virtual picking proxy above the button ui itself.

[knotty dune]

Note that for all my hoverable widgets, I don't depend on enter/leave events. Instead I poll the HoverMap resource: https://github.com/viridia/bevy_reactor/blob/main/crates/bevy_reactor_obsidian/src/hover_signal.rs#L11

[summer talon]

yeah, that's probably advisable. enter/leave is just polling the hover map under the hood, and it's less flexible.

[knotty dune]

Because the HoverMap resource changes every frame (or at least, it did with bevy_mod_picking), I use an intermediate component to store hover state. This computes "hover within" meaning that it's true if hovering over any descendant.

[summer talon]

interesting.

i think there's some stuff i can do with that.

you've given me an idea

[knotty dune]

This means that the widget only reacts when the hover state for that widget changes, not any other widget.

And the reaction is just bog-standard component change detection

@stoic flicker I am somewhat chagrined to say, my Pressed and Hovering components are in some ways a re-invention of Interaction. Although without the roll-off bugs 🙂

[stoic flicker]

Yes :p

[stoic flicker]

Feel free to PR to my PR?

I'll swap to HoverMap tomorrow if needed

[summer talon]

won't have time to do this before release

[stoic flicker]

kk

I'll do the HoverMap change tomorrow then, and put this in Draft for now

[summer talon]

i think hover-within should be exposed as a first-class concept, I think we can expose it as part of the observer api.

probably captures what most people mean by "hovered" better

[knotty dune]

It's very similar to the :focus_within() selector in CSS - which triggers whenever any descendant has keyboard focus.

[summer talon]

let me cook on that for a bit maybe i can get something like that landed in the next cycle

[knotty dune]

@stoic flicker @summer talon @tepid trout I've thrown together a small PR which introduces a new crate, bevy_input_focus, which defines the basics of keyboard focus and bubbling keyboard events. https://github.com/bevyengine/bevy/pull/15611

[thick umbra]

How opinionated can I be about the mesh picking backend / how much can I change relative to bevy_mod_raycast? I could just make a PR that's a direct port with minimal changes (I have it technically done) but there's a ton of tiny things I would like to change or simplify

mainly around inconsistent or confusing naming, what data is stored and in what format, also the actual AABB ray cast is sub-optimal

and API things

[summer talon]

you should be very opinionated imo. we only need the code directly used by the back-end, you can discard the deferred api entirely i think.

might be nice to fix the ray-cast inefficiency upstream if you can.

but we can do that later probably.

i'd prefer if you did two PRs if you want an actual overhaul: one to do the minimal upstream and one for refactoring.

it's much easier to review that way.

[thick umbra]

sure, I can try... I'm just having trouble deciding what I should vs. keep or rework to minimize controversiality

[summer talon]

yeet anything that's easy, leave in the non-trivial cases initially and we'll refactor it later.

[thick umbra]

Like do I keep RayCastPickable (other picking backends don't have a similar marker afaik), do I keep SimplifiedMesh, do I keep backface culling configuration, do I change types to use what Bevy would normally use (like Dir3 for normals)

[summer talon]

first pr should be a direct subset of the original with bits missing, ideally. We can hash all this out in a very opinionated follow-up.

[stoic flicker]

I spent a bit testing out a few options for hovering strategy, and went with a simple "check all descendants against the hover map" approach: https://github.com/bevyengine/bevy/pull/15597

I think that we should be okay to merge that (and I would like to do so promptly to avoid running into the required_components PRs)

[summer talon]

checking descendants every frame is probably the best option. As an alternative we could count the number of pointers hovering it's descendants (over is increment, out is decrement, could possibly parallelize it with atomics) but we'd need to add some kind of gross change detection on parent/child to make it robust to changing hierarchies.

[stoic flicker]

You could collect the set of buttons into a hashset or two, and then iterate upwards

But yeah, much messier

[summer talon]

You only need to propagate when the focus map changes, so it could be significantly more efficient

If we want to make a general utility for this, it should probably work that way, but that’s a problem for 0.16

[stoic flicker]

Yeah, and the hover map list should be a lot smaller

But yeah, I so don't care right now

[knotty dune]

When you say "check all descendants", you don't mean walk the tree downward, right? What I do is start with the current hover entity and walk up until I find a 'Hovering` component, so it's O(N), not O(whatever).

[stoic flicker]

I'm doing the former right now. I wasn't convinced of the correctness of your code just reading it, but I can change to the upwards approach before merging if you think this is important

[knotty dune]

Well, a linear search is going to be cheaper generally: the cost will be proportional to the length of the ancestor chain. The worst case is when hovering over nothing, so you have to go all the way to the top. But this isn't usually very long, even the most complex UIs tend to have a depth on the order of 10 or so.

Yes, buttons typically have few children, and this is true of most hoverable widgets, but I can't say for sure that there won't be a hoverable widget with a large and complex sub-tree.

[stoic flicker]

Yeah, I definitely think it's more algorithmically optimal!

I just found the iterator chaining in your implementation harder to read, and less "obviously correct"

(despite probably being correct)

[knotty dune]

Let me look at it, it's been a while

[stoic flicker]

Feel free to PR to my PR with an optimized form too 🙂

[knotty dune]

So we're talking about the line parent_query.iter_ancestors(*ha).any(|e| e == entity)

[stoic flicker]

Yep 🙂

[knotty dune]

Actually, let me give the whole context:

let is_hovering = match hover_set {
    Some(map) => map.iter().any(|(ha, _)| {
        *ha == entity || parent_query.iter_ancestors(*ha).any(|e| e == entity)
    }),
    None => false,
};

The reason for the outer loop is because the hover map has multiple pointers; the logic here is "it's considered hovered if any pointer is hovering over the entity"

Note that for some reason I did it differently for the focus case, where the code looks like this:

fn is_descendant(world: &World, e: &Entity, ancestor: &Entity) -> bool {
    let mut ha = e;
    loop {
        if ha == ancestor {
            return true;
        }
        match world.get_entity(*ha).map(|e| e.get::<Parent>()) {
            Some(Some(parent)) => ha = parent,
            _ => return false,
        }
    }
}

I don't know which one is faster, it depends on how much overhead there is setting up the Query for Parent.

[stoic flicker]

what does ha stand for here? You really love you abbreviated variable names :p

[stoic flicker]

Great, apparently RelativeCursorPosition is a thing, which I need to find a replacement for in this PR, since it was tightly coupled to Interaction 😅

Oh and we need to kill FocusPolicy too

[summer talon]

why is this needed and can it be a custom QueryData that wraps Pointer?

[stoic flicker]

Custom widgets like sliders

But yes, that's my hope too

[summer talon]

i feel like sliders should just calculate this for themself. part of the difficulty of this migration is that bevy was not designed well for touch or multiple cursors.

RelativeCursorPositon won't work for touch.

[stoic flicker]

Yes, but we have a slider in our examples 😅

[knotty dune]

Yeah, I should have picked a better name there, and I'm not sure why I chose that particular abbreviation, usually I'd name it something like "search". I tend to follow the conventions in "the practice of programming" book, in which short names are acceptable for variables with extremely small scope, like i in a for-loop.

The reason for that variable was to avoid mutating the function parameter, which I tend to avoid for stylistic reasons.

[stoic flicker]

Yeah, I tend to really be on the "descriptive names" side of that arbitrary style fence

[halcyon pilot]

Do we have a solution for passing picking events through an image used as render target? :o
This would be useful for #editor-dev

[summer talon]

https://dev-docs.bevyengine.org/bevy/picking/pointer/struct.Location.html

[halcyon pilot]

Oooh, nice

[summer talon]

you just need to make a custom pointer driver, as by default they only are added to windows.

[halcyon pilot]

I'm inserting Pickable on a NodeBundle and observing Pointer<Down> and getting nothing, is that expected? I can only seem to get Pointer<Drag> events

[summer talon]

mm that's strange

can i see some code?

you don't need to put Pickable on stuff for it to be pick-able.

weird to get a drag without a down though

[halcyon pilot]

https://github.com/tim-blackbird/bevy/blob/editor-areas/examples/tools/editor_areas/main.rs#L192

[summer talon]

Btw @stoic flicker in 0.16 i'd like to bring the pointer event api further into convergence with the w3c Pointer Events spec. It has a verity of additional useful sounding stuff we could steal very directly without changing the lower levels at all.

winit is already moving in the "follow the w3c spec" direction for keyboards so i think there's precedent. thoughts?

[halcyon pilot]

... Wait, oh no. It was my own fault. I was using print instead of println. How silly

[halcyon pilot]

Well at least we got some value out of me being silly :p https://github.com/bevyengine/bevy/pull/15630

[halcyon pilot]

Thinking about it again, isn't the Pickable component really confusingly named? Pickable seems to imply quite strongly that it is necessary for something to be 'pickable'
I doubt I'm going to be the only one with this misconception

[thick umbra]

(I thought it was necessary to add too 👀)

Oh it just overrides default picking behavior? That's confusing

[halcyon pilot]

The simple_picking example also adds Pickable :p
Whoever made/reviewed that were also confused

[fluid sandal]

https://github.com/aevyrie/bevy_mod_picking/blob/main/examples/render_to_texture.rs

Pickable used to be mandatory, and became optional with the addition of backends. It's a byproduct of iteration.

The name is definitely misleading.

[halcyon pilot]

I thought I remembered it being mandatory at some point, I think it's due for a rename before we ship 0.15 then. Thanks

[stoic flicker]

Issue please

[stoic flicker]

And for this 🙂

[halcyon pilot]

Typing one up now

[stoic flicker]

You're the best

[stoic flicker]

https://github.com/bevyengine/bevy/pull/15707

[fluid sandal]

FYI, I think the perf of mod_raycast is more than sufficient, it's handling raycasting against caldera hotel with no sweat. When I'm not render-bound I'm hitting the 120fps vsync cap. If there is any question about perf for the mesh picking backend, I'd say just yoink all the code and AABB acceleration stuff I did, it seems to scale well in real scenes (many low/med-tri objects).

[stoic flicker]

How do y'all feel about bumping Interaction-yeet to 0.16? It's a lot more complex than I expected, and I'm nervous about trying to cram it in

[summer talon]

i'm fine with that.

it's probably one of the easier things to bump

and will let us get a nice api for hovering children

[stoic flicker]

Yeah

[stoic flicker]

Hey @thick umbra what are you plans for the mesh picking backend?

Still feasible for 0.15?

[thick umbra]

Was busier than expected with uni stuff for a few days, but I've had more time today and can try to get a PR ready today or tomorrow

[summer talon]

please don't crunch, but do hand it off you can't get it all the way there

[thick umbra]

I don't think I can have the picking backend in bevy_mesh after all :/

It'd have to depend on bevy_picking, which depends on bevy_render, which depends on bevy_mesh again, and that's a circular dependency

ig I'll have to use my original approach of having a separate crate

I feel like it'd be much easier to define all the first-party backends in bevy_picking directly though, behind feature flags, it'd avoid these weird dependency shenanigans

third party backends could still be defined externally

[summer talon]

the dependency on bevy_render can be removed i believe.

[thick umbra]

it's used for the Camera

[summer talon]

mmmmm drat

[thick umbra]

and render targets

[summer talon]

i am tentatively ok with this solution

[halcyon pilot]

Looks like it could be done if we extract the concepts of render targets and viewports from bevy_render.
bevy_picking requiring bevy_render isn't great (long-term at least IMO)

[summer talon]

yeah i've been trying, it's a very hard knot to cut

i don't think we should try to attack that coupling before 0.15 release

but i do want mesh picking in for 0.15

[humble fog]

Yeah, while it's not ideal to have the bevy_render dep, the mesh picking backend is way more important

[thick umbra]

I'll probably put the mesh picking backend in bevy_picking behind a feature flag then, and open another PR to move sprite picking and UI picking there as well? Or I could have that separate bevy_mesh_picking or bevy_picking_mesh crate like I have right now, it just adds to our crate debt and is inconsistent with the other backends

[humble fog]

I think it's fine to just have all of them in the picking crate for now

[stoic flicker]

This won't work for bevy_ui, since it'll create a circular dependency

[humble fog]

oh

[stoic flicker]

We want to be able to respond to picking events in bevy_ui eventually :p

[thick umbra]

So... a separate crate for mesh picking after all? Or some weird mix where mesh picking is in bevy_picking but the others are elsewhere

[stoic flicker]

I think a seperate crate is the least bad immediate choice.

[thick umbra]

kk good, that's the easiest too since I have it already :P

did we have an opinion on bevy_mesh_picking vs. bevy_picking_mesh?

-# (naming, the most important issue ofc)

[humble fog]

could it just be a single bevy_picking_backend crate?

I'm not sure where the circular dep gets introduced

[summer talon]

simplest solution: just throw the mesh picking impl in bevy_picking for now and we'll move it somewhere better later

i really don't want to introduce a crate for just one release cycle.

[thick umbra]

fair

[summer talon]

behind a feature flag, maybe. but honestly almost everyone who uses picking is going to want that.

at least we can make good use of the rendering dep while we have it.

[stoic flicker]

Yes please to feature flag 🙂

[thick umbra]

I'll have a PR ready tomorrow, need to head to bed now. Here's the WIP branch though for the curious, or in case I happen to vanish into thin air
https://github.com/Jondolf/bevy/tree/mesh-picking

(I'll still change and improve some things there but it's functional)

[flat sky]

Personally disagree with this sentiment. For me personally I'm very judicious about which backends I enable

Would appreciate the feature flag

[fair raptor]

UI crates will want picking, but not mesh picking

Unless it doesn't pull in a ton of the rendering crates, in which case a feature isint really needed

[summer talon]

Picking itself (unfortunately) depends on our bevy_render.

The feature will probably only gate a dependency on bevy_mesh.

[thick umbra]

Okay, here it is!
https://github.com/bevyengine/bevy/pull/15800

great color contrast and readability on the label when hovering over it in the existing simple_picking example

[stoic flicker]

Graphic design is our passion

@tacit tulip @sharp dock, present for you 😉

[tacit tulip]

what am i looking at

[stoic flicker]

An extremely clear, legible example to teach people how to point at things

[thick umbra]

also is it intended that Gabe is hella blurred in the sprite picking example

needs some nearest neighbor interpolation on that one

[tacit tulip]

-# Click Me to get a box

[thick umbra]

these examples are... something

[tacit tulip]

beautiful.

[stoic flicker]

ooh MSAA strikes again

[thick umbra]

Why does the sprite picking example even have all this animation and like, a grid of birds? We don't even skip transparent pixels for the picking, and the bounding rectangles for the Bevy birds are really confusing

also...

why is Gabe not running

??? why is the second sprite tiny in the sprite_animation example on main

(but not on website version)

oh it doesn't have a transform

[stoic flicker]

Maybe.... scale factor accounts for the discrepancy? That's bizarre

[thick umbra]

I see

yeah our main branch is a train wreck lol

[stoic flicker]

We'll get there 🙂

Soo many painful migrations this cycle

[tacit tulip]

guh

[thick umbra]

Ah, it was still querying for TextureAtlas, but for sprites, that should now live on Sprite directly or it won't work

This is another Handle-like situation; that's extremely breaking with zero errors or warnings

[stoic flicker]

I'm looking forward to removing the Component impl from TextureAtlas too 😄

We need to give UiImage the Sprite treatment still

[fluid sandal]

It was added to test all the wacky combinations of sprite offsets and transforms

It should really be a test though

It's just easier to set up in an example, though it doesn't seem appropriate to upstream.

[thick umbra]

yeah it makes sense as a test

[fluid sandal]

Awesome. Thanks Jondolf!

[thick umbra]

No problem, thank you for making the ray casting and picking stuff in the first place 😄

[fluid sandal]

I don't know how some people maintain OSS libraries for more than a few years. I'm sick of thinking about picking. 😄

[fluid sandal]

reminds me, something I've wanted to do for a while, but haven't had the time, is set up a proper headless testing harness with mod_picking, so we can test this stuff in CI end to end. You can drive virtual pointers without the need for winit/etc, and validate that they are hitting or not hitting targets as expected. It should be pretty simple to get something set up, but will take a bit of effort to start building all the test cases.

It should be as simple as spawning the scene, sending an input event, running the app one update, and checking for a hit event.

[thick umbra]

(fixed these examples https://github.com/bevyengine/bevy/pull/15803)

Where should I put the reflecting_laser example? It's not necessarily a picking example

just 3d?

[stoic flicker]

either 3d or picking

Probably 3d

[thick umbra]

also should I call it reflecting_laser or something like mesh_ray_cast, probably the latter?

[stoic flicker]

It's not actually picking at all

[stoic flicker]

latter please

Also we could put it inside a d20, just saying

[thick umbra]

a 20-sided die?

an icosahedron

need to add Platonic solids to bevy_math first /s

[stoic flicker]

Just add RegularPolyhedron(usize) obviously

[fluid sandal]

I think that's an icosphere with zero subdivision?

The cube is cool because it alternates between order and chaos at a few points.

No idea what an icosahedron would look like, might be cool though!

Or you could fill a hand authored mesh like Suzanne with lasers 😆

[turbid arch]

Looking forward to mesh picking (and all the other picking on 0.15).
Do we have an example that tries various ways of pick->hold->drag-to-move?
This scenario seems prone to be buggy due to small things like ordering, and (literal) edge cases like moving outside the window etc.
If we don't have it I think it should exist.
Was this ever a source of issues in your experience @fluid sandal ?

[fluid sandal]

I didn't use it much tbh, because of the ordering issues. That should theoretically be fixed now though.

[knotty stone]

I feel very much attacked 😬

To my defense, I just needed some way to test that the code I copy-pasted actually worked end-to-end. So more of a "technically this works now" than a proper example...

[knotty stone]

Is that the bevy_editor_cam? that cam looks amazing

[fluid sandal]

Yup!

[knotty stone]

to me it looks like the panning is relative to the object that is under the cursor? is that correct?

[fluid sandal]

Yes

[knotty stone]

very cool

[thick umbra]

Did you happen to have this caldera hotel demo with picking public somewhere? I'm kinda interested in comparing perf traces for bevy_mod_picking vs. what I have now

[fluid sandal]

Griffin has some glbs here somewhere

[thick umbra]

(made ray-AABB intersection tests much faster, reduced allocs, etc.)

yeah found it
https://github.com/DGriffin91/bevy_caldera_scene

[fluid sandal]

#1159383661062389790 message

There's the same glb

Make sure you are using the no skel version

[thick umbra]

thanks!

[fluid sandal]

Nice

[deep rock]

why not consider Pickingbehaviour as required component instead of optional component? Taking sprite as an example, if game contains thousands of sprites but few of those are pickable, picking_backend will traverse all of those with much unnecessary cost.

[summer talon]

It was previously and it would be easy to treat it as required.

Just add it to everything

[fluid sandal]

That component isn't needed, and requiring it is a waste of memory. That change was made for a reason. It was common for some types of picking to want picking on everything by default, like UI.

I'd say this feature probably belongs on the backends themselves. That, or an opt-out zst for everything, but the cost really only exists in the backend implementations themselves.

[spice carbon]

I think it's used in LWIM to detect if UI intercepts input. Do we have another abstraction from bevy_picking for it?

[stoic flicker]

🤔 I think we need focus management to do that properly

DreamerTalin's been working on ideas, and I'd like to spin up a working group for it in 0.16

Ideally bevy_egui can hook into it too

[spice carbon]

Totally agree, but if we just remove it without a replacement, it will break user games :(

[stoic flicker]

Yeah, which is why I'm not shipping it in 0.16

In 0.15, sorry

[knotty dune]

One thing that should be relatively easy is "pluggable focus navigation": different games and different platforms will have different methods of moving the focus between controls.

• Tab-navigation like on the web, with support for modals
• Gamepad using direction buttons
• Graph-based
• Coordinate-based
• Optionally integrated with LWIM ("Focus Up", "Focus Left" commands)
  These can all just be top-level input handlers that are last on the bubbling food chain.

[devout hollow]

Sorry to ask in the dev chat, is the picking API going to be part of UI behaviour at all in 0.15?

[stoic flicker]

Nah that's fine. The answer is that it works with UI (basic functionality) but the existing solution will coexist for a release

Which will give users time to migrate and engine devs time to refine

[devout hollow]

I had seen the 0.16 migration plan which is why I wanted the clarification, thank you ^^

[halcyon pilot]

PointerInteraction is missing pub accessors that were present in bevy_mod_picking, (https://docs.rs/bevy_mod_picking/latest/bevy_mod_picking/pointer/struct.PointerInteraction.html)
bevy_editor_cam uses those.

Context: I'm trying to add bevy_editor_cam to the bevy_editor_prototypes repo, which requires porting it to Bevy main

[halcyon pilot]

hmm, well I guess I'll make a PR to add those

[halcyon pilot]

https://github.com/bevyengine/bevy/pull/15964

[halcyon pilot]

It looks like UI elements don't block lower nodes from being picked, regardless of what should_block_lower is set to

In the following example the root node always recieves Poiner<Down> events. I can prevent it by adding trigger.propagate(false) to the child

It's very strange because the root node does produce Pointer<Out> events when you go over the child, but doesn't produce Pointer<Over> when you hover back over the root node

No wait, it is even buggier than that

Oooh, its because traversal/propagation is turned on that it's doing this. That is very confusing. I think we don't want this

Not for UI at least

This is icky

Yeah, this makes no sense for UI, especially the Over and Out events getting propagated. Here's another example, see for yourself what happens

[stoic flicker]

Yeah, issue please!

[halcyon pilot]

Ugh, oh no, this is how html does it too, but html also has a `mouseenter' event that does more what I'd expect

[halcyon pilot]

So there is no way to model a hovered state with observers and pointer events at the moment, we need a 'mouseenter' equivalent

Well, I guess I can add PickingBehavior::IGNORE to the children in this case

[summer talon]

This is a known issue

I don’t think I can resolve it for 0.15

I’m future I’d like to hue much closer to the web spec

[stoic flicker]

Yeah the web spec for this seems genuinely good

[knotty dune]

The way I handle this in quill and bevy_reactor is that I don't use mouse enter/leave events, I use the hovermap and ancestor traversal

[fluid sandal]

This is correct

[stoic flicker]

https://github.com/bevyengine/bevy-website/issues/1723

[gaunt hemlock]

does the new picking allow to pick without triggering the original picking events? I mean like in web dev tools where you can pick any dom element but the pick won't trigger the picked element events

I was thinking how a similar feature could work in Bevy, with a separate "debugger app":

• the debugger app tells the running game/app to enter "dev picking mode"
• the user switches to the game, can pick ui elements/meshes/… (with hover feedback)
• the picked element details show in the debugger app (not sure how the game communicates this to the debugger via brp… maybe the initial BRP call just waits until the pick; kind of a "long pool")

[halcyon pilot]

I think you can do this by looking at the HoverMap directly

[fluid sandal]

Not generally, no, but it could be added to the bevy_ui picking backend pretty easily, I think.

[summer talon]

You could also just toggle the event dispatch plugin part on and off.

[fluid sandal]

Was just looking to see if that was possible, I don't think it exists in the 3rd party plugin

I don't see a way to do it in the upstreamed version either. We would need to have a run condition on the update_interactions system, I think.

Maybe pointer_events too.

Either way, shouldn't be too hard to add. You want to build the hover map, but not send any pointer events or update interactions.

[fluid sandal]

As for BRP, the hover map is just a resource, so as long as you can read a resource and query the ECS, that should be doable.

[summer talon]

I've identified a few things I'd like to ship in 0.15 to smooth the transition for mod_picking.

• Exposing picking events through EventReader again. This is absolutely essential for upstream crates and our work at foresight. Should be trivial, PR will be up soon.
• Window picking: picking events bubble to the window, pick the window entity if nothing else is hit. Not essential, but nice to have for UI. PR will be up soon.
• Add a field to Trigger that indicates the original entity targeted (when bubbling) and a field that indicates the depth of the traversal (starting at zero and incrementing). Not started on. Important for building stuff on top of picking.

I know we are trying to slim down the features remaining for the release, but I think these are all pretty uncontroversial and won't break existing functionality. So I'd really like to sneak them all in during the RC.

[summer talon]

Window picking should be ready for review: https://github.com/bevyengine/bevy/pull/16103.

[stray frost]

https://github.com/bevyengine/bevy/issues/16065#issuecomment-2438900604

I think for 0.15 we need to reconsider our defaults. But we should also rethink the "mesh picking" scenario entirely as I think its current applications are extremely limited

(reply on github if you have thoughts)

If this has already been discussed somewhere, let me know. I know I'm hopping in late. Just reacting to the current state of main / whats currently slated to land in 0.15

[summer talon]

that link does seem like a real issue. I do think the current mesh picking is great for the editor and camera effects (mesh picking drives foresights bevy_editor_cam which is pretty well regarded, and can also be used for camera-focus/DOF stuff). Benchmarking shows this performs surprisingly well.

I wouldn't call the applications "limited". There are specific but not uncommon, and users may sort of expect a system that just works for those specific cases.

[thick umbra]

That aligns roughly with what I've been saying from the start in terms of what I'd expect ray casting to be like. I found it strange to have this kind of mesh ray casting by default, typically it should be done with colliders and a BVH acceleration structure, not globally on raw mesh data. The performance cost is arguably quite minimal in profiles though, since we're typically only casting one ray per frame with the mesh picking backend, so that part probably isn't a huge concern

[summer talon]

It seems valuable to provide a way to "picking style" trigger observers for click and hover events (ex scenarios like "I want to write an observer for click/hover events when a 3d player collider is hovered"), but I think we need "scoping" to be an integral part of the system.

I agree, but I also feel like this is exactly what the current system is designed to provide. You do

world.observe(|trigger: Trigger<Pointer<Click>>, player_coliders: Query<&PlayerColider>| {
    if let Some(colider) = player_coliders.get(trigger.target()) {
        // add my logic to coliders here
    }
});

[humble fog]

(reply on github if you have thoughts)
😅

[stray frost]

Yup lets keep the convo there

[summer talon]

right, right, sorry. I'll re-post some more considered thoughts there

In other news, Picking Event Streams https://github.com/bevyengine/bevy/pull/16105 should also be ready for review.

[fluid sandal]

@thick umbra I couldn't help myself, it's a 7LOC addition

.observe(on_drag_rotate)

//...

fn on_drag_rotate(drag: Trigger<Pointer<Drag>>, mut transforms: Query<&mut Transform>) {
    let mut transform = transforms.get_mut(drag.entity()).unwrap();
    transform.rotate_y(drag.delta.x * 0.02);
    transform.rotate_x(drag.delta.y * 0.02);
}

[thick umbra]

Ooh cool! That might be good for release notes

along with the fancy laser example to showcase ray casting specifically

[fluid sandal]

I can't help but love the generality/composibility of this observer. Any entity that is pickable and has a transform can be rotated when dragged.

oh my god

it works on the UI node

this is incredible

#general message

[stoic flicker]

What PR is this?

[fluid sandal]

I was working on some small cleanup to some of the picking and UI examples

No PR yet

[stoic flicker]

Okay sweet. Ping me when it's up 😄

[fluid sandal]

I wanted to say the results of the upstreaming are great. Thank you to everyone for the hard work, I really appreciate it. It's a huge relief knowing I can get this off my plate.

Love the ergonomics of the observers too! It's great to have the dream of event listeners realized, but less scuffed! 🙂

[knotty dune]

@fluid sandal Something I've been considering is the process of migration from bevy_mod_picking to bevy_picking. My engine currently has a lot of mod_picking code in it, and it will be easier to migrate features incrementally rather than all at once, which should not be too hard so long as mod_picking and bevy_picking are both able to co-exist, which is true today. However, I don't know what the plans are for mod_picking supporting future bevy releases.

[fluid sandal]

https://github.com/bevyengine/bevy/pull/16129

[fluid sandal]

Not completely certain at the moment. @summer talon has a migration guide already prepared, I believe. We'll need to take a closer look as part of the 0.15 RC.

If the upstreamed version covers our cases, and the 3rd party backends move into their respective crates (rapier, avian), there is a good chance we do a minimal update for compatibility, but mark the plugin as in maintenance mode, on track for archive.

[summer talon]

The two versions should be able to coexist but you’ll have to be careful not to mix the types up. Maybe we can unify some of the types to avoid that issue in the 0.15 legacy version of mod_picking?

[wintry spoke]

With the new stuff is there a way to universally filter out things like 'middle mouse drags' ? At the moment I get a drag for any mouse button, and it seems like I need to filter these out in every observer.

[summer talon]

currently, I don't believe there's a better way to do this. We def should fix that. For the next release I want to look at some gentility functions for building observers and adding filters like this.

btw wonder if you have thoughts on https://github.com/bevyengine/bevy/issues/16065#issuecomment-2438900604

cart def seems like he thinks there's room for improvement on the picking front end api

[summer talon]

I'm working on the upgrade path for mod_picking, the plan is to release a version for 0.15 with a legacy feature that preserves the old core and eventlistener stuff. Without the legacy feature, all the backends and selection/highlight will target the upstreamed bevy_picking and the old core/input will not be exported.

I think that's the best we can do, in terms of offering a smooth upgrade path. Let me know if you have any other opinions on ways to make things easier.

[wintry spoke]

I don't use mesh picking, so I haven't seen errors.

Cart's explanation matches what I'd expect for 3d, in that picking would usually either be against a simplified mesh, a physics mesh, or a two step process of checking against a volume and then optionally some kind of more detailed shape. (This is what I've done in Unreal, back in the day, or in other situations where picking against a complex mesh is too expensive).

For 2d, which my game is, generic box-like detection works for basically everything. I can't, at the moment, pick against spine since i resolve the bone positions (skin) on the gpu and have no way of reading that back. So even for characters I do box detection for selection. Tiles, etc, are also fine with the current API.

At one point I was selectively enabling picking backends based on the gameplay context, but I ended up not needing that.

The main issue I run into is the need for filtering. For example, almost all my picking has a carve out for "are we over egui or does egui want the mouse" (hope to get rid of egui once a native inspector is around). I also run into things like "i want to filter out all non-primary mouse buttons for this interaction" like I mentioned above.

What aevyrie says also makes sense. Anyway, I feel like I should stay out of this one. You all are more than capable of a good resolution...

I may be wrong, but I sense the tension here is between "having a comfortable default that isn't a foot gun" and "having a very generic, flexible underlying system." It doesn't see like these need to be in conflict, but the tools for physics shapes don't exist yet.

[summer talon]

cool thanks, that's good feedback. I'll think more about filtering, I do think that plays into what cart is saying a bit.

he wants us to have ways to filter and send different events to different observers, rather than having to filter in the observers.

[wintry spoke]

Yeah, I'd like that as well.

In many cases I've run into, most of my observer code is filtering.

(Where the observer just needs to do something simple in response to a trigger.)

[summer talon]

checks out

as complexity increases you need to go back and additional filters to existing observers too, i imagine

[wintry spoke]

However, overall, triggers and observers are a huge triumph. We are using them all over now. They help with ordering (which is guaranteed), they let outside systems react to commands without the command needing to know about the external systems, so overall they just beat events in many cases.

(You can have a command trigger something, then check to see if the trigger did something, etc)

But my game is very UI-like (its a dwarf fortress style game) so I'm not surprised my use cases aren't problematic for what's being built. It's simple stuff. 😄

[fluid sandal]

That egui problem is handled by having an egui backend. If egui reports hits the rest of your picking code doesn't need to have any knowledge of egui.

The picking system will just see the egui hits on top, and will block things below.

[fluid sandal]

Another note, physics collider based picking backends already exist, but they have to be out of tree obviously.

[thick umbra]

I would expect 0.16 to get primitive shape ray casting as well (#15724) which you could build your own "collider" types and ray casting backend for. This of course involves lots of DIY before first-party physics though, but could maybe be useful for projects that don't need/want a full-blown physics engine

[fluid sandal]

It would be really interesting to hear how you would expect this filtering to look.

Oops, meant to reply to @wintry spoke

[wintry spoke]

I have no idea! I'll think about it. My immediate ideas don't make much sense, like add_observer but with a run_if that can be systemlike or something.

[fluid sandal]

We could probably make some sugar wrappers for pointer events specifically, that you can chain together, that would add a filter in front of your code

[stoic flicker]

What if we had a filter generic in Trigger?

I've wanted that before for sure

[wintry spoke]

Yeah adding the filter on the trigger call would work

[stoic flicker]

This also lets us bail earlier

[summer talon]

This seems like something @stray frost does have thoughts/plans for.

[fluid sandal]

Sounds cool to me

[fluid sandal]

Reminds me, I wonder if Interaction should become an observer as well, for best of both worlds. Coalesce the interaction state for the user and only pay for when it changes.

This might even be preferrable to all the individual pointer events.

e.g. Interaction would store information about any state change for over/out, drag, enter/leave

[stoic flicker]

👀 Wouldn't you need to store the state somewhere to get a cohesive view?

[fluid sandal]

Only when it changes. Right now everything gets it. We could maybe use the observers themselves somehow? Haven't thought it through.

Like, you could store state about interaction in the interaction observer itself somehow. So, the only entities that store the state are the ones listening for that state.

It would need to be accessible from outside the observer though.

A dumb way of doing it would be Observer<Interaction> has Interaction as a required component

But that is basically just component change detection at that point.

[stray frost]

From my perspective the tension in that conversation came from misunderstandings, which I think was largely my fault due to poor framing in my first message. Based on the issue @fluid sandal filed in response, it seems like we're all on the same page now.

I do think the current picking impl is the right general fit and I don't think theres any immediate pressure to refactor into something to cover the more generic cases

[stray frost]

Open to the idea, although given that its largely an ergonomics problem being solved, it would need to be able to express the filter ergonomically. Given that the filter would be providing "value matching functionality" (ex: I want a specific enum variant), I think that might require const generic enum support to do ergonomically (which afaik hasn't landed yet). And any cases not covered by that would require the matching function to be defined outside of the Trigger type and then passed in via a generic, which would be less ergonomic than doing it in the observer

[stoic flicker]

I don't think we necessarily need value matching functionality for this to be useful

[stray frost]

Could you write what the Trigger code would look like?

[stoic flicker]

Before:

fn cast_spell_on_enemy(trigger: Trigger<Pointer<Click>>, enemy_query: Query<(), With<Enemy>>, mut commands: Commands){
   if enemy_query.contains(trigger.entity){
     commands.add(DoSpellThings{trigger.entity()})
   }
}

After:

fn cast_spell_on_enemy(trigger: Trigger<Pointer<Click>, With<Enemy>>, mut commands: Commands){
   commands.add(DoSpellThings{trigger.entity()})
}

[thick umbra]

That specific case already works with just Trigger<Pointer<Click>, Enemy>, right? You just can't do e.g. Without<T> or other more complicated filters, and tuples of components have that unintuitive "or" behavior

[stray frost]

Ahhh I thought we were talking about filters on the Pointer event, not filters on the targeted entity

[stray frost]

This is expressible, although idk if it actually functions as intended in this case. This reads as Pointer<Click> fired for the Enemy component target, not an entity that happens to have Enemy

[stoic flicker]

It's not clear if the existing B generic works like this for non-lifecycle observers

The docs about what that generic does are super lacking

[thick umbra]

Yeah I just tested it and it doesn't work, the event is never triggered if you have that generic

there goes my intuition about how observers work again 🙃

[stray frost]

You need to explicitly trigger the event with the entity and component target pair

[summer talon]

Yeah, components passed to triggers are treated as context applying to the event, rather than modifying the target.

[stray frost]

Ex: this is what the "insert component event for entity " observer trigger looks like:

[summer talon]

OnAdded requires a component as context, stuff like that

[stoic flicker]

TBH, I'd rather swap to an OnAdd<T> API

And lose that generic completely on Trigger

[thick umbra]

Yeah it feels wrong that component targets seem to be purely an internal feature (the APIs are private from what I can see) but the generic is there for all triggers

[stray frost]

We might encounter some issues here in the context of the more complicated observer scenarios as it combines the target component into the event component, essentially erasing the target component from the API. Just an intuition though, I don't have those scenarios easily retrievable in my brain atm 🙂

[stoic flicker]

Makes sense 🙂 I'll try and take an earnest look at this in 0.16. Just wanted to make sure we were all in agreement on "ideal API"

[stray frost]

Might still be able to use that API at the surface level, if we're willing make the typing more complicated (ex: don't accept a target event component directly, instead accept an TargetSource or something that can feed into both contexts)

No strong opinions atm. Just need to make sure any changes account for the "whole picture"

[summer talon]

i think @valid pulsar once gave me a good reason to not do this

[thick umbra]

Nevermind, apparently you can give component IDs to trigger_targets #ecs-dev message

so you could use the generic for custom events too

this really needs to be documented, I've never seen or heard about this

the docs on Observer just say that events can be triggered for entities

[valid pulsar]

Once you move the generic on to the event you need to consider how to invoke the trigger dynamically. Currently OnAdd has an id and each component has an id so it's pretty straightforward. To support OnAdd<MyComponent> it would have a different type id to OnAdd<OtherComponent> so you would likely need to implement some trait that allows you to pass through the necessary ids. It also means you can no longer look for all events of a type regardless of component, which isn't very relevant for OnAdd as registering an observer for all adds is terrible for performance but for custom trigger types you may want to see all of them, you could support that with OnAdd<()> or some other marker type if you wanted.

I'm sure you can do it, but it's not trivial without losing some functionality (although it may be functionality that no-one is using as it's not well documented)

[spice carbon]

Is there a way to make all entities non-hoverable except for a few others?

Ah, depends on the backend, I see!

[stoic flicker]

We need release notes for this work 🙂 I think I can write this up, but I'll need reviews

[summer talon]

I will have time to review and to contribute to the notes.

I have a very detailed draft migration guide too, I’ll update that and post it here

[stoic flicker]

Fantastic, that'll be a big help

[stoic flicker]

This is a bit beyond what I want to call Trivial, so I'd really appreciate a second review on https://github.com/bevyengine/bevy/pull/16229

[summer talon]

Should be the last little item needed to more or less fully replicate what was possible in bevy_mod_picking.

[summer talon]

oops no i lied https://github.com/bevyengine/bevy/pull/16231

i said i would change this ordering ages ago and forgot to do it

[fair raptor]

Need hover events 👀

Not just Over, but also enter/exit

I had some manual stuff in bevy dioxus

Also to kill off Interaction

[stoic flicker]

On my docket 😄

[summer talon]

Planned for early in 0.16 absolutely

[fair raptor]

Btw, when do input observers fire?

If I click on an entity, when do I expect to see the observer trigger?

Because ideally I want it right at the start of the frame, before a system that reads the state my observer is modifying

And ideally there shouldn't be any frame delay between winit registering a click, and the observer firing

[summer talon]

PreUpdate I believe.

Winit registers clicks outside the normal ECS schedule

We need to run systems to determine pointer hover, and then to trigger the actual events

That happens later in the frame, but it should always happen in the frame the event is exposed by winit.

Yeah, all the events are triggered by ‘pointer_events’ system in pre update

[fair raptor]

Hmm, I'm getting a lot of input lag between click UI -> UI is visibly updated

Maybe it's just my imagination?

[summer talon]

Platform? It’s possible this is an issue with the winit event loop. I don’t think it’s even possible for picking to cause lag.

[fair raptor]

If you run https://github.com/bevyengine/bevy_editor_prototypes/pull/145, drop a GLTF onto the window, and then click the text rows in the scene tree.

[fair raptor]

Windows

[summer talon]

I’ll try when I get a moment later, I don’t get this on the engine picking examples, could you try to replicate on them?

[fluid sandal]

This list is basically the plan.

[fair raptor]

At work sorry, can't

[fluid sandal]

Enter exit is non trivial because it requires more hierarchy traversal. It also might be the case that we do need to keep the interaction component around to be able to aggregate state across pointers, but it's not clear yet.

[summer talon]

Same, I’ll investigate though, thanks for flagging it

[summer talon]

There’s some rumblings about caching ancestor lists for transform propagation

We can maybe exploit that.

[fluid sandal]

Cool.

[fair raptor]

Don't we need traversal anyways for bubbling... oh that hapens at the observer level

Bevy dioxus didn't have this issue, because I didn't use observers and wrote my own bubbling and event dispatch system.

[summer talon]

There’s discussion on approach a bit higher up in this channel

[fluid sandal]

You need to be able to tell if the entity or any children in the subtree are hovered. Could get expensive fast if we aren't a tiny bit clever.

[fair raptor]

Taking picking events -> calling the write entity handlers

[fair raptor]

This is how I did it: https://github.com/JMS55/bevy_dioxus/blob/main/src/events.rs#L165-L195

HashSets

[fluid sandal]

That's essentially the same idea as the hovermap and previoushovermap, except it needs to handle multiple pointers.

[fair raptor]

Yeah I'll say I haven't followed any of this closely, I may just be repeating old discussions 😛

[fluid sandal]

I think the only tricky bit is deduplicating the events across pointers, and not repeating a bunch of extra traversal work. It shouldn't be too hard.

I'm tempted to try a naive solution today if I have the time. Enter/exit would make reactive style updates so much easier.

[knotty dune]

What I do currently is that I have a marker component which indicates that a widget is interested in knowing about hover states - meaning that it wants to know if itself, or any child, is being hovered. I then have a system which runs every frame, looks for those markers, and polls the hover map. It then updates the hover state information on the entity.

[stoic flicker]

Parallel convergence on the same ideas is a good sign 😄

[knotty dune]

Actually it's not a marker, it contains a boolean which is whether the entity is being hovered.

But I can use change detection on that component to decide when widgets need to be visually updated.

[fair raptor]

After thinking about it more, I think it's actually due to styling.

The highlight effect I have only takes place after you release the click button

I think if I had a "is clicking" style, and added some animations, it would feel much nicer

[summer talon]

Ah yeah, click is a very late event, in terms of where in the state machine it fires.

[knotty dune]

What happens if you change it to pointerdown instead of click?

The main reason the click event exists (I'm speaking of the web here), as opposed to just pointerdown/pointerup is to allow cancel via roll-off (I clicked but changed my mind, so I can roll off before I release the button). However, for widgets which take no action other than selection, there's no reason why we have to wait for pointerup/click to change the state - because the action is easily reverted by clicking again. So checkboxes, selected rows, etc. can use pointerdown without harm.

[fair raptor]

Idk, I can test later. But I would probably look at existing apps I have and check what their behavior is, and then copy it.

[cedar hill]

Something I don't really like about the current picking event system:

Notice how hover exit emits when the cursor move onto the text? I don't think that should be the default behavior.

[summer talon]

Yeah, this is the over/out behavior jasmine was talking about. I just added docs about this, it’s because picking is not hierarchy aware

We have plans to add enter/exit events that track hover for all children

In 0.16

Unless @stoic flicker wants this prioritized for 0.15, lots of people do seem to be hitting this

[cedar hill]

btw is there an event for mouse down inside then up outside? (cancel is not it it seems)

[summer talon]

I don’t follow

[fluid sandal]

Unlike the web, bevy's hierarchy places the text as its own node. For now, you likely just want to make the text Pickable::IGNORE.

Until we have proper enter/exit events.

[fluid sandal]

Doesn't exist, no.

Though you could check manually with the dragmap iirc

You could also just look at DragEnd and check that the target isn't the same as the listener

[summer talon]

If you get an Out after a Down I guess?

Instead of a Click

[fluid sandal]

This is just a special case of a drag end as far as I can tell

[cedar hill]

well you can technically move out then back in then click 😆

[fluid sandal]

Sorry, I misspoke, DragDrop.

I guess that wouldn't capture the case where you fully leave the window or aren't over something during the release

I would still probably look at DragEnd, because it should handle that case. You just need to check if the pointer is over the original entity when the drag end is received. If it was, it's a click. If it wasn't it's a cancel.

But that should be pretty easy to add if this is a standard event. It would be added in the same place we send the Click event, but it would be the opposite branch.

[summer talon]

I can add that.

But I feel like it would be more correct to use Enter/Exit for this (when they exist) or Over/Out (for now).

[fluid sandal]

Yeah, I'm not familiar with this event. A cancel has different semantics: https://developer.mozilla.org/en-US/docs/Web/API/Element/pointercancel_event

I'd prefer to not expand the set of events to maintain unless it already exists in the js pointer API.

enter/exit/scrolling are probably the most useful ones we are missing right now.

[fair raptor]

Imo text should be pickable ignore by default

I always forget to do that

Btw @summer talon if you're adding more docs, can you clarify in the module docs that unlike most events, picking events are propagate by default, and you should explicitly stop propagation in the observer if needed

That confused me a lot last night, until I realized where it was configured (in the event jmpl)

[summer talon]

Oh yeah, should do.

[summer talon]

I guess this is possible with required components

[stoic flicker]

I'm reluctant to add it 🙂

[summer talon]

Yeah, we have too much already, but people are going to ask about it the entire cycle lol

[stoic flicker]

Yeah but bevy_mod_picking didn't have it so 🤷‍♀️

[cedar hill]

the only thing I used in bevy_mod_picking was Interaction 🙃

[turbid arch]

Not sure if this is the place for it, but I'm wondering @fluid sandal since some amount of mod picking is in main now, what will happen to your plugins which use your own mod picking for 0.15?
Will you still use your own internally or will you migrate?
(I use stuff like bevy editor cam etc.)

[fluid sandal]

plan is to migrate

[turbid arch]

Oh cool! I wasn't sure enough was in main yet.

[fluid sandal]

everything should be in there

The backends in the 3rd party plugin will be moving to their respective crates. E.g. the physics backends will be adopted by their respective bevy integrations.

[summer talon]

As requested by @knotty dune, It will be possible to use the original eventlistener based picking on 0.15 with a feature flag, but by default (at least in my local migration) everything except highlight/selection/debug is getting removed, and those crates are re-targeting the upstreamed version.

excepting some possible bugs and namespace conflicts, it should be possible to use both at the same time, and have a nice gradual migration

[fluid sandal]

The entire plugin, including highlight and selection plugins will likely be archived, I don't have any desire to maintain them.

(at work, we will just vendor the current versions, and tweak them for our use case)

Selection will likely have some similar focus abstraction in bevy_ui. Highlighting will probably need similar things in bevy ui as well.

[summer talon]

i have a full migration of the crate almost ready, just need to update it to the newest rc

[fluid sandal]

might want some sort of "interaction component updated" observer to make it easy to update entities based on the final computed interaction state that frame

[old venture]

I've fiddled the ray_mesh_intersection implementation and expanded the benchmark suite. I reran the numbers multiple times and they look okay but I'm on a m1 mac laptop, ideally I'd like someone on a different OS and/or not a laptop to make sure the results are repeatable. Pr here 🙏 cheers 🙂 instructions in the pr about how to run the previous impl with new bench suite

[stoic flicker]

Draft release notes are here: https://github.com/bevyengine/bevy-website/pull/1806

In the meantime, the simple_picking example (at least) seems broken

Wait no it's not

I was wrong

[stoic flicker]

Okay, the example works!

I love these changes for UI

So so much better

[fluid sandal]

once we get enter and leave it will be way way better

and/or Interaction improvements

observers/listeners make retained UI so much nicer

[stoic flicker]

I intend on landing both 😄

We're going to make bevy_ui actually nice damn it

[summer talon]

i have a branch with enter/leave almost done

thank you for writing this btw, sorry I didn't get to it in time

[fluid sandal]

Reviewing it now.

Thanks Alice

[summer talon]

btw https://github.com/bevyengine/bevy/pull/16231 is fixed and should be ready for review again

it's pretty small, issue was just a doc comment

would be good to get in for 0.15

[stoic flicker]

CI appears to be failing ❤️

But yes, strongly agreed

[summer talon]

oh duh, i conflicted myself

[summer talon]

all good now

if you add/remove a child that is hovered, you need to get a enter/leave, I don't have that implemented yet but the core is done.

[stoic flicker]

Sounds good, we can add that later

[fluid sandal]

finished the review, thanks again for doing the hard work 🙂

[stoic flicker]

Are you sure you submitted it? 🙂

Oh wait on my PR!

Not nth's

[stoic flicker]

I snuck in a bevy_event_hearer joke 😄

[flat basin]

Is the upstreamening complete as of 0.15 or is there more upstreaming to be done in the 0.16 development cycle?

[summer talon]

the up-streaming is complete, bevy_mod_picking will likely be archived in the near future.

but there is still more work to do

testing, refining, improving the ui integration