Better Audio

[dusky mirage]

Ok, I implemented Diff and Patch for usize and isize! https://github.com/BillyDM/Firewheel/commit/271724baa76e220307f631e1de9d26904d4e1b94

[limber kernel]

ah okay i think i was getting hung up here - so i wont need RealTime clone or anything with patch_skip?

[dusky mirage]

Correct. If you are not using the patching system, then you don't need it to be clonable.

[rapid hedge]

thanks!

[dusky mirage]

@slate scarab Do you think your freeverb node is ready to be merged? Or did you want to do some benchmarking first to see if using #[inline(always)] and using unsafe indexing into the delay line makes a difference?

[slate scarab]

Oh I think the performance is probably good, I could run like six of them at once on a microcontroller (same implementation) with no issue. So it should be okay for a 100x more powerful processor haha.

[rapid hedge]

Also, tiny bump re: https://github.com/CorvusPrudens/firewheel-web-audio/compare/master...dur-fix 😉

[slate scarab]

ya once we get everything releasing, I'll make another for this crate and the ircam hrtf one

[rapid hedge]

I did 👀

[slate scarab]

once we have a profiling setup, i'd be happy to give these optimizations a shot

[rapid hedge]

ready to try this now

do I need to insert AudioEvents myself?

[slate scarab]

it's a required component on all nodes

[rapid hedge]

ah, neat

[rapid hedge]

which of the 3 node entities do I push on?

[slate scarab]

hm, could you remind me what they are again 😅

[rapid hedge]

One in the pool, one per sampler, one in the graph

or something like that

[slate scarab]

It's fine if you give it to the ones related to the SamplePlayer, it'll be moved over to the real one.

And then flushed to the graph.

[rapid hedge]

and I should be able to get those via....

let me think

get_effect_mut gives only the component, not the Entity

[slate scarab]

you could just get_effect with a query that has entity

[rapid hedge]

that works? 👀

neat!

[slate scarab]

oh right but if you're pushing directly it doesn't matter

you don't need the entity

[rapid hedge]

Well at that point I can just query the events, yeah

neat!

So many things to discover in seedling!

[dusky mirage]

Hmm, do you think this has something to do with workspace_dependencies?

[rapid hedge]

I would have hoped not

since your usage of workspace deps looks very similar to how I do it in rerecast

And that would mean rerecast could also pull in bevy

[slate scarab]

This is how it's supposed to work anyway, I haven't tested with non-cloneable events yet. So if it doesn't work, let me know! I'll push a patch real quick to handle that.

[rapid hedge]

Sure!

I'll go eat now and then clean the rat cage, then I'll give it a try

Look at this stinky lad

[slate scarab]

stimky

[tropic bison]

Ahh that’s awesome to hear! Thanks for clarifying

[steep dove]

it's clearly on track to become the default audio engine, but making that offical is going to take some time

[tropic bison]

How good is firewheel when it comes to audio timing precision btw?
From my experience bevy audio is very imprecise with playback timing, and I think the root of that is rodio and scheduling stuff. Does firewheel have some improvements in this area?

[slate scarab]

Unless you schedule things too soon (as in the moment has passed by the time an event reaches the audio engine), you'll get sample accurate timings for all events.

(By default -- you could relax this if you want.)

[tropic bison]

Ah that sounds amazing!

I’ll try making the switch over the next few days. Thanks again!

[slate scarab]

The latest version is still on the rc, but that's compatible with the latest Bevy release.

[rapid hedge]

So I have this query: mut ambisonic_node: Query<(&mut AudionimbusNode, &mut AudioEvents)>,
And I call this:

let (mut node, mut events) = ambisonic_node.get_effect_mut(effects)?;
events.push(NodeEventType::Custom(OwnedGc::new(Box::new(
    "hi".to_string(),
))));

Then in the processor, this reports 0: info!("Number of events: {}", events.num_events());

is that an upstream oops?

[slate scarab]

The error there could just be that we don't transfer those kinds of events. You could try placing it on the actual node by following the Followers relationship on that entity.

(It's a collection, but in practice there's only one follower -- the real node that's in the pool.)

mut ambisonic_node: Query<(&AudionimbusNode, &Followers)>,
target: Query<&mut AudioEvents>,
// ...

let (node, followers) = ambisonic_node.get_effect(effects)?;
let mut events = target.get(followers[0])?;

You shouldn't need to do this of course, but that would verify that it's an issue with moving around the non-clone events.

[limber kernel]

One other thing - is there a way to make sending an event "nicer"? It looks like NodeEventType::Custom accepts any Owned data, so I send a new IR with calling into, to make it easy

impl Into<NodeEventType> for ImpulseResponse {
    fn into(self) -> NodeEventType {
        let b = Box::new(Some(self));
        NodeEventType::Custom(OwnedGc::new(b))
    }
}

Idk why but this feels strange to me - I want to do something like define a ImpuseEvent enum or something and abstract all the type stuff away from the end user. Is that possible or orthogonal to the design of the custom event/owned? (i guess it always needs to wrap itself in an OwnedGc?)

[rapid hedge]

followers[0]? Surely you mean followers.iter().next().unwrap()

[rapid hedge]

doesn't look like it

[slate scarab]

oki i will fix in like 30 minutes

[limber kernel]

#[diff[skip]]
pub not_diffable: NonCloneableType

still gives
the trait bound NonCloneableType: Clone is not satisfied the trait Clone is not implemented for NonCloneableType

[slate scarab]

(did you mean diff(skip)?)

[limber kernel]

huh, i wonder how that worked prior. yeah, i did - still same error though

[slate scarab]

Oh I think I see the issue. While the operation is skipped, the macro is still applying trait bounds.

[limber kernel]

ah that would line up

and to clarify, i should be able to derive clone on the node still with this skipped, right?

[slate scarab]

Okay I take it back -- it doesn't do this. It should behave correctly.

[slate scarab]

Well if it's not cloneable, how could you derive clone on the whole type?

I might be missing some context.

[limber kernel]

No, I think you're right

i dont have any formal experience with programming so if it seems like im doing something incredibly weird it probably is just that lol

[slate scarab]

oh haha no worries
i haven't followed this issue to closely -- hopefully there's a nice solution here

[limber kernel]

yeah i think i just dont derive clone on the type because, as you said, obvious reasons

for whatever reason i thought it was a requirement for nodes

[slate scarab]

It is for bevy_seedling 😅

[limber kernel]

ohhhh

[slate scarab]

We were actually discussing how to solve this for audionimbus earlier. Basically, for something like this which is produced once and then sent to the audio processor, imo it's better to keep it out of the AudioNode struct. That causes a lot of headaches! You do lose the data-driven aspect, but imo it's not too hard to write an API that allows users to trigger this sending event.

[rapid hedge]

yeah this sounds very familiar hehe

[slate scarab]

In the ECS at least, you can quite easily completely abstract it away from the user.

Outside of it, there ought to be some nice ways to encapsulate the behavior there.

[limber kernel]

got it, that makes sense! im wondering for something like this - where does that abstraction layer live? Should it be a part of the nodes in firewheel under a bevy feature gate, in seedling, or left up to the end-user to implement?

im guessing the last one is probably not a good solution, and there might be other solutions im not thinking of

[slate scarab]

ya that's the tricky part

This is related to the PR on Firewheel, right? Is the problem obvious from the code that's there currently? Or is this followup work?

[limber kernel]

yep! I was trying to implement a field on the node, but i think this convo dissuaded me from that. my main qualm now is sending the event

[limber kernel]

so like id imagine on the bevy side of things that there would probably be a trigger or something i could use to pass in audio

[slate scarab]

You might also load it from an asset

[slate scarab]

Like for example you could have an ImpulseResponse(Handle<AudioSample>) component that just constructs this event as soon as the underlying asset loads.

[rapid hedge]

mind pinging me when you did? 🙂

I think I'm blocked until then

but no rush!

(still cleaning the cage async from coding huehue)

[slate scarab]

commands.spawn((
    ConvolutionNode::<2>::new(...),
    ImpulseResponseAsset(server.load("some_impulse.wav")),
));

fn send_ir(
    mut nodes: Query<(&ImpulseResponseAsset, &mut AudioEvents)>, 
    assets: Res<Assets<AudioSample>>,
) {
    for (ir, mut events) in &mut nodes {
        let Some(asset) = assets.get(ir.0) else {
            continue;
        };

        events.push(ImpulseResponse(asset.clone()).into());
    }
}

[slate scarab]

ya know something like that

(obviously you'd need additional checks for whether a particular asset has already been sent, etc.)

Outside of Bevy, I don't think it's too bad to have users generate these events themselves. Presumably, they'll have ready access to the IR assets.

[limber kernel]

oh yeah that would make more sense to do it with components

that seems like something seedling level, maybe? or is it still kind of an open question

[slate scarab]

Yeah this suggestion would require work in seedling, but given the node is being merged into Firewheel, I think that makes sense. At least if you agree with the direction.

[limber kernel]

yea def. just wondering order of operations basically - so i can tidy things up, get it merged in firewheel, then can add some better ergonomics to seedling (if im interpreting correctly)

completely different question: it isn't guaranteed that a declicker will settle within the same buffer it was enabled/disabled in, right?

[slate scarab]

ya, that's correct

it is not guaranteed to happen

[limber kernel]

would be nice to just fade out of one IR, fade in the other in the same processing block. but i think i gotta store the old IR data until it finishes transition

[slate scarab]

While declicking generally won't last too long, there's really no minimum size for the processing block. It could be quite small.

So I think that's unavoidable.

[limber kernel]

i keep seeing buffers with like a million floats and thinking "wrow thats so many system resources" but in actuality its like as much as a single texture (tho idk about how things actually look in memory)

[rapid hedge]

I think 1 MiB of RAM is doable if you're not on a GBA lol

corvus typing

is it a fix

[slate scarab]

no lmao

Do you have this in-progress work available btw? I'd love to work with it for validation.

[rapid hedge]

me or doomy?

[slate scarab]

u

[rapid hedge]

can push it

sec

pushed

it's nothing except that one string event haha

[slate scarab]

nice

oki well this'll do it

for patch in events.drain_patches::<AudionimbusNode>() {
    self.params.apply(patch);
}

info!("Number of events: {}", events.num_events());
if let Some(NodeEventType::Custom(event)) = events.drain().into_iter().last() {
    let string = event.get().downcast_ref::<String>().unwrap();
    info!("Received custom event: {}", string);
}

i suppose the name drain_patches can be misleading

[rapid hedge]

what's the difference to now?

the code looks identical

[slate scarab]

To be clear, drain_patches will drain all the events and discard anything that's not a NodeEventType::Param. So running that before checking for the custom events will always remove any custom events.

[rapid hedge]

Oooooooh

I thought you pointed out the code that fixes it

but you pointed out the issue!

[slate scarab]

ya just a sec

[rapid hedge]

oh yeah of course

it's less "drain patches" it's more "drain all and gimme the patches"

[slate scarab]

ya

[rapid hedge]

So something like

for event in events.drain() {
    if let Some(patch) = AudionimbusNode::patch_event(&event) {
        self.params.apply(patch);
    }
    if let NodeEventType::Custom(event) = event {
        let string = event.get().downcast_ref::<String>().unwrap();
        info!("Received custom event: {}", string);
    }
}

yep that works!

[slate scarab]

ya basically, i was cookin up this

for event in events.drain() {
    match event {
        NodeEventType::Param { data, path } => {
            let Ok(patch) = AudionimbusNode::patch(&data, &path) else {
                continue;
            };

            self.params.apply(patch);
        }
        NodeEventType::Custom(custom) => {
            let string = custom.downcast_ref::<String>().unwrap();
            info!("Received custom event: {}", string);
        }
        _ => {}
    }
}

[rapid hedge]

now let me see if this works without the Followers thingy

[slate scarab]

dang it doesn't

but that's what I expected after looking at the code

[rapid hedge]

nopes

Looks like the ReflectionEffectParams still need to be behind an ArcGc<OwnedGc>

because each node gets the same one in this case (it's calculated once for the single listener)

And Cloneing it would duplicate that sneaky non-Sync pointer

[slate scarab]

do you need it twice though?

[rapid hedge]

WDYM?

[slate scarab]

ReflectionEffectParams should only have one node to be sent to.

It should be produced once in the ECS for exactly one audio processor

[rapid hedge]

shouldn't it need to be processed by every source in the scene?

[slate scarab]

hm, for the one from the listener?

There's two, right. One ReflectEffectParams is used for reflection calculations, and this is calculated by each source. But then there's one from the listener's source, and that's only calculated once.

[rapid hedge]

Yep, exactly

But it's applied to every source:

for the source's reverb

[slate scarab]

ew

[rapid hedge]

which means every processor needs its own copy

(I'm assuming we have one processor per node)

[slate scarab]

The only way to allow that to be lockless would be to store it in the Firewheel processor itself in a type erased store. But we don't have that at the moment. I mean maybe ArcGc<OwnedGc<T>> would compile, but that doesn't seem sound.

[slate scarab]

And then each node could look into that store and try to fetch the reverb.

When it comes times for their audio processing.

[rapid hedge]

So I need to send the effect in an arc<owned<T>> event for now, right?

or is checking a lock during a processor a no-go

[slate scarab]

Nah locking is definitely better than potential unsoundness.

[rapid hedge]

aight

[slate scarab]

But the other one doesn't need this treatment.

It can just be sent as an OwnedGc

[rapid hedge]

well the event is behind an OwnedGc

So it's really an OwnedGc<ArcGc<OwnedGc<T>>>

which goes in an option

[slate scarab]

good good yes

[rapid hedge]

a good old Option<ArcGc<OwnedGc<T>>>

Rust moment

[slate scarab]

perfection

[rapid hedge]

Can I get an owned value out of the event?

[slate scarab]

you could mem::swap

[rapid hedge]

nvm figured it out

[rapid hedge]

nvm

[rapid hedge]

audionimbus::SimulationOutputs doesn't have a default

so uh

I guess I can send an Option in the event

to I can .take it

[slate scarab]

yes

just one more wrapper

[rapid hedge]

but would be neat if there was a real API for this

[rapid hedge]

@drowsy dome would say

 if let NodeEventType::Custom(mut event) = event {
    if let Some(update) = event.get_mut().downcast_mut::<SimulationUpdate>() {
        if let Some(outputs) = update.outputs.take() {
            self.simulation_outputs = Some(outputs);
        }
        self.reverb_effect_params = Some(update.reverb_effect_params.clone());
    }
}

[slate scarab]

beautiful

[rapid hedge]

@cold isle there's a little soundness issue in audionimbus

it's fine if you're careful, but technically...

you can create as many copies of ReflectionEffectParams as you want by calling reflections repeatedly

and send each copy to an own thread

And when accessing them there, you violate !Sync

[slate scarab]

okaay @rapid hedge a cargo update bevy_seedling should allow you to drop the Followers trick

[rapid hedge]

thx

[rapid hedge]

OwnedGc contains an ArcGc

so, does ArcGc<OwnedGc<T>> even make sense?

[slate scarab]

I believe OwnedGcWrapper uses a bit of unsafe to assert its properties, and the inner ArcGc isn't shared anywhere except to the garbage collector itself

so, yeah i think so

well

i mean no, it would be like hoping Arc<Box<T>> would allow you to mutate T

[rapid hedge]

hmm I see

Followup

[slate scarab]

for this use case, i think you'll have to ArcGc<Mutex<T>> it

[rapid hedge]

but that's bad, @dusky mirage said 👀

[slate scarab]

until we allow you to store that data in the graph anyway

[rapid hedge]

because it allows the OS to park

[slate scarab]

Yeah but it's less bad than unsoundness

In practice, you won't actually have terrible performance (see rodio). It's just not ideal.

[dusky mirage]

What's the problem?

[rapid hedge]

Can I get a &mut T from an ArcGc<OwnedPtr<T>>?

[dusky mirage]

Hmm, why is the OwnedPtr wrapped in an ArcGc in the first place?

[rapid hedge]

need to clone it

to share it across processors

[slate scarab]

Here's the context: several nodes need access to a single type that's !Sync. It could be made Sync by putting it in a Mutex and sharing that, but obviously that's not ideal.

It would be great if you could store such types in a type-erased store within the FirewheelProcessor. Then, each node could get non-overlapping mutable access to such a type without any locks.

Other folks have mentioned wanting something like this. And we have a clear motivating example here.

[dusky mirage]

Oh, ok. Well Firewheel is single-threaded only, so I guess a Mutex probably won't ever trigger an OS park.

But yeah, having a type-erased store in the Firewheel processor itself seems like a more elegant solution. That should be pretty simple to add.

[rapid hedge]

I s'pose I'll use the Mutex for now then

[slate scarab]

I think you could use this to send arbitrary data between nodes as well in user-space. For example, MIDI processors could push to and pop from a type they have in the store.

Which I know a few folks would really appreciate.

[dusky mirage]

Oh, true.

[rapid hedge]

Then try_lock should also be good, right?

Since it cannot really be accessed in another thread in practice

(it's only ever used in processors)

[dusky mirage]

Yeah, try_lock could work too.

[rapid hedge]

yep it works 🙂

[slate scarab]

I think it would basically be similar to how nodes can each have their own state -- but instead of being able to store a single dyn Any per node in the FirewheelContext, we'd want to be able to store an arbitrary number of dyn Any in the FirewheelProcessor. And then I guess the ProcInfo or some other parameter would give access to this store. Maybe ProcExtra actually.

[rapid hedge]

Oh hey I can actually get around this

I don't need a &mut access

(I added the &mut in the wrong place lol)

[dusky mirage]

What do you think would be the best API for such a type-erased store? Just a global HashMap<u32, Box<dyn Any>>?

[rapid hedge]

@slate scarab IIRC the issue we had with the buzzing was that the effect was used twice, right?

not the effect params

[dusky mirage]

Although maybe we should have a better system to avoid collisions than just a u32 ID.

[slate scarab]

Probably like Resource in Bevy. i.e. only one item of each type is allowed. We can key it to the TypeId.

So a get would be O(1)

[dusky mirage]

Oh, interesting.

[slate scarab]

This is relatively elegant because it allows good composition.

[dusky mirage]

How do I implement such a thing? Is there a crate I can use?

[slate scarab]

If you want to guarantee no collisions for some piece of state, just create a private type and store it. On the other hand, if you want to interact with state from other crates, you know how to get it; the type is the key.

[rapid hedge]

I now made the apply method on all effects take &mut self @slate scarab @cold isle. That seems to be more accurate to what Steam Audio is doing internally, namely mutating that effects object

[slate scarab]

Bevy's impl is a little too bespoke for what we'd need. But it should be quite simple, the basic idea is this:

struct Store(HashMap<TypeId, Box<dyn Any + Send>>);

impl Store {
    pub fn insert<S: Send + 'static>(&mut self, state: S) {
        self.0.insert(TypeId::of::<S>(), Box::new(state));
    }

    pub fn state<S: Send + 'static>(&self) -> Option<&'_ S> {
        self.0.get(TypeId::of::<S>()).as_deref()
    }

    pub fn state_mut<S: Send + 'static>(&mut self) -> Option<&'_ mut S> {
        self.0.get_mut(TypeId::of::<S>()).as_deref_mut()
    }
}

Maybe panicking impls are better by default

[dusky mirage]

Ok, and what about no_std support? Does alloc contain a HashMap?

[slate scarab]

bevy_platform does

[dusky mirage]

Ok, cool!

Hmm, though adding new entries to this store will allocate. But I suppose if it's only for the first initial block, then it might not be a big deal.

Oh, I suppose I can add a way for the user to modify the store before the Firewheel processor is constructed.

[slate scarab]

Maybe we could reserve space for particular keys when initializing the stream or just after

Each slot could be an Option<Box<dyn Any + Send>> maybe

[rapid hedge]

@slate scarab mind giving https://github.com/MaxenceMaire/audionimbus/pull/20 a review?

[rapid hedge]

Also, I believe https://github.com/MaxenceMaire/audionimbus-demo/pull/2 is now sound

[slate scarab]

oh yes i will review in the morning!

[drowsy dome]

firewheel may have cpp bindings in the future? :o

[slate scarab]

oh no you're diving head first into flecs arent you xD

[drowsy dome]

dude

it's rocked my world

like im a bit stunned. my original game plan from a year ago was to build a rythm-based soulslike, akin to fury

i have decent progress on it, but realized that to build much beyond the viewport

you've got to be jan hohenheim

and i am not jan hohenheim, im dscallops

idk. but the extensive ability to query for things, systems that are all too common, buildable via the flecs dsl is just insane

that's the theory, at least. ive not put it into practice

[slate scarab]

okay make a pilgrimage to flecs and please come back one day and help improve bevy_ecs

[drowsy dome]

all i wanted to do corvus, was to collapse a double for loop smh. i think ive gotten way ahead of myself here. also I've still got to work on my shader, sfz, and like 10 other things. trying my hardest to spread thin

like a watery hummus paste

[slate scarab]

trying my hardest to spread thin
mm yes as the programming gods intended

[dusky mirage]

Alright, I added a ProcStore type for node processors! I even made it so the user can access it through FirewheelCtx if the stream is not currently running.

Unfortunately it did add a breaking change to node processors, but I have added a new context that should hopeful prevent further breaking changes.

[slate scarab]

oh we might be able to give it a spin with audionimbus immediately

[dusky mirage]

Man, I remember thinking about a month ago that firewheel_core was pretty much complete. 😅 Though it's good that we are working out all of the pain points before it officially lands in Bevy.

Did we need to make any modifications to the Diff/Patch macros, or are we good there?

[slate scarab]

hm, well aside from this Send, !Sync stuff (which i think is kind of irreducible), there's nothing off the top of my head

jan did mention that it would be nice if you could easily move stuff out of the NodeEventType::Custom variant

[dusky mirage]

Oh right. That should be easy to add!

[slate scarab]

ya im very glad we got some movement on integrating something like steam audio (thanks jan :3)

[rapid hedge]

thank you

seriously, you carried me through this haha

[rapid hedge]

docs that #[diff(skip)] exists 😉

[dusky mirage]

Actually, it looks like I need to change Custom(OwnedGc<Box<dyn Any + Send + 'static>>) to Custom(OwnedGc<Option<Box<dyn Any + Send + 'static>>>), to make this work.

[rapid hedge]

at least https://docs.rs/firewheel/latest/firewheel/diff/trait.Diff.html and https://docs.rs/firewheel/latest/firewheel/diff/trait.Patch.html should mention it imo

[rapid hedge]

yeah that's what I ended up doing too

[dusky mirage]

Ok yeah. I'll make that change on my end then.

[rapid hedge]

flecs is hella based

honestly wish Bevy would just copy-paste it

For audionimbus + AI, I'm questioning whether I want one or two simulators

On one hand, the expensive simulator runs for the player anyways, so might as well use it

On the other hand, users might want audio quality settings to tune according to their hardware, and we wouldn't want the NPC AI to behave differently when you change your sound quality

so I'm thinking maybe there should be a second simulator, sharing the same scene

I was also thinking about some other optimizations I could do, and I'm now tending towards duplicating the audionimbus stuff in the ECS for the NPCs

which should be fine, it's not much boilerplate

i.e. the player's audio could use the hypothetical steam audio plugin, and the NPCs use a custom implementation that is fully inside the ECS

[slate scarab]

ya you should be able to make it so much cheaper that two simulators is worth it

[rapid hedge]

exactly, and the NPC simulator can be super low quality

That approach immediately shows that an audionimbus integration plugin should expose the scene as a resource or something, so that the bespoke NPC code can reuse it

and the Context ofc

For simplicity, I'll start with the assumption that the steam audio plugin uses exactly one simulator

Any additional simulators are the user's business

so that means that the SimulationSharedInputs are global

Corvus, I remember you saying that you had an idea how to deal with that

was it a system that went and rebuilt all nodes when the resource changed?

probably by changing their config component?

[slate scarab]

yes that sounds correct

[rapid hedge]

Do you have an opinion on how the user-facing API should look?

[slate scarab]

hmmmmm ill probably have more opinions after various reviews in the morning

[rapid hedge]

I'm wondering if there even are any per-source settings

[drowsy dome]

a question about ::Custom variant of anything. kinda a throwback to axum honestly with how they handle state. It's not at all the same thing as an audio node, I know. However, there is a similarity here. where y'all have nodes, axum has services via tokio tower.

[dusky mirage]

Alright, I added a NodeEventType::downcast method!

[drowsy dome]

now, here's the thing

[rapid hedge]

oh yeah these

[rapid hedge]

thanks a bunch 😄

[drowsy dome]

so for example, me passing midi stuff around via firewheel is fine, downcasting and whatnot

but I think steam audio here and myself were kinda facing similar issues

[rapid hedge]

oh hey you'll also want the shared state thingy

[drowsy dome]

a strongly typed variant, but i mean, it's fine if it's not

[rapid hedge]

#1236113088793677888 message

[drowsy dome]

it's just like, the variants of events are difficult for me to get along with, but ill get over it eventually hehe

if there's no other way, there's no other way

but tower is remarkable

[slate scarab]

what jan linked to was essentially "resources-in-processors"

[drowsy dome]

the idea of it that was.

[slate scarab]

so instead of only events, you can coordinate arbitrary communications between nodes, all strongly typed

[drowsy dome]

maybe we're saying the same thing. in theory you can do that now with the custom

i don't know why it bothers me 😅

but if I had it my way, AudioNode<T> would be the trait def

[drowsy dome]

are we talking about https://docs.rs/firewheel/0.7.2/firewheel/event/enum.NodeEventType.html

I'm talking about this

[slate scarab]

no different mechanism, but i see what you're talking about

[drowsy dome]

where any instance of this now is replaced by a generic T

this enum can never be exhaustive in theory

[slate scarab]

yeah the event storage would be a bunch of type erased stuff, and then each processor would unerase it according to what it expects

[drowsy dome]

I see. I would have assumed, for example, some node for steam audio would accept event T where T: SteamEvent or something idk

or even a common trait exported through firewheel at least

[rapid hedge]

why tho

[slate scarab]

okay maybe i don't see what you mean 😅

[drowsy dome]

because it means that the user defines the domain of their graph and, as a result, their program need never consider invariant cases

[slate scarab]

i dont think that composes well

if i understand the idea

imo each node needs to be atomic -- imposing or receiving constraints in a viral way would basically kill the ecosystem

[drowsy dome]

but the point here is that it's preventing virality

that having something here like NodeEventType requires all users consider every variant, which can never be exhaustive. On the contrary, if someone makes a node for uh, logging for example, you need not have to implement their trait nor include it in your graph

but it also means now firewheel doesn't need to add a monad when everyone asks for a logger

take the midi variant of this type

who's node will be utilizing data of this variant? midi consumers. the solution of course, is feature gating this variant. and it is

but now looking at CustomBytes...hmm, and Custom?

most people require () here

[slate scarab]

The design is a trade off. The most common cases are encoded directly in variants. Floats, integers, and so on. Then, in the slower, dynamic variants, you can encode anything.

Aside from that, I really don't see how you could possibly have a composable ecosystem with this generic approach. Every time you add some third party crate, you have to find a way to implement its trait. That would be unbelievably unergonomic. (Again, if I understand the idea.)

Part of how serde is so flexible is due to its data model. It has a kind of intermediate representation that is simple, yet you can represent any Rust type with it. The NodeEventType is Firewheel's data model. It facilitates effortless composability (from the perspective of Firewheel crate users).

[drowsy dome]

that's true, because its data model reqiures that to be hehe. it's literally ser/de. firewheel is not a serializer and should not be tell users which custom events they condone. When there are specific use cases for a package that requires data chained through multiple graphs, they're probably doing unnecessary type reflection in their nodes. this is partly why ive found it easier to pass channels around. But in this scenario, this tolerance of arbitrary events kinda ends at firewheel. In some respects, I agree with what you're saying. The same case cannot be that's not necessarily true for seedling (maybe the minimal setup would definitely be true). However, downstream seedling libraries would most likely opt to implementing their own trait on a node type of seedling's choosing, or require that users implement some type.

the orphan rule flows both ways here

crate authors that fail to adhere to a certain standard will not capture packages. and if they do, then it must be because of something of value for their consumers, who would be willing to constrain their own graph to the usage of said crate

by requiring the nodeventtype to be of one and only one set of variants, determined by an author who need not make any use of the type, it chokes the ability for users to do special things and well-formed ecosystems. because remember, that downcast is still gonna work even if two crates that share data don't sync their deps correctly

[dusky mirage]

Well, the main thing is we want to avoid excessive allocations/deallocations. The vast majority of events will be simple primitive types. Custom events are mostly just for passing custom resource data to nodes.

[slate scarab]

Hm, I'm not sure I follow.

it chokes the ability for users to do special things
You can represent anything with it. What do you feel it's missing? Efficiency? Compile-time correctness?

I guess I'm wondering what the core objection here is.

[drowsy dome]

I think it's both of those, efficiency, compile-time correctness, but also variability. I couldn't possibly know all of the things that are possible with a generic event type on nodes of course. I'm still on that lil 1 credit audio course (sick btw). But it's also interpretation of data, and whatnot. what happens when two crates decide that array means different things for example? well, idk. i made that situation up in my head, it doesn't exist! so it's not even a problem yet hehe

However, at least from what I do all the time, the work stuff, that custom state capability is freakin awesome. freakin FromRequestParts is so good, for example. if I had to be given a slab of bytes and then figure out wha tmy api's state is every time
edit: https://docs.rs/axum/latest/axum/extract/trait.FromRequestParts.html what a beaut. type enforced data on the api handlers. wish i had an example on hand

axum would be so dead rn

but also if something else interpreted my state due to the lack of type enforcement, I'd have to figure out what they want without expression of their trait

[dusky mirage]

In fact, my original plan was to have just a single normalized f64 and an integer parameter ID for all parameter events (this is what DAWs do). And node authors would spin up their own message channel if they wanted to pass resources to their node processors. But since then we added more and more features to make it more friendly to data-driven frameworks like Bevy, and it since evolved into its own complex event handling system that also automatically schedules events.

[slate scarab]

Okay but this is bar for bar Patch, no?

NodeEventType is RequestParts. Patch constructs a type-safe message from the intermediate representation.

[drowsy dome]

well, hmm. would you Patch things to send messages down a node graph?

I don't know honestly, i wouldn't think so though

[slate scarab]

Are you talking about inter-node communication?

[drowsy dome]

I am, across different crates and whatnot

audio pipelines >:)

[slate scarab]

No, the purpose of Diff and Patch is to communicate state from a source and to a destination that a single author controls. Attempting to interpret the intermediate representation itself would be a little bit like peering into a type's raw bytes on the stack. Brittle.

[drowsy dome]

hehe that's my point! hopefully I'm not sounding too loopy

because I'm dealing with this already in my own roundabout way

if any crate wants to build off mine, then that crate author and I have to literally communicate to coordinate how we're going to handle our channel

like on discord

[dusky mirage]

Well, the aim of Firewheel is to be like wgpu but for audio (as well as a bunch of optional factory nodes that covers the essentials). People can create whatever system they want on top of it, there aren't strict guidelines that nodes have to follow.

[drowsy dome]

makes sense, I mean I could create a node that has a channel receiver and broadcaster

[slate scarab]

Right, so what's one way we communicate between Bevy systems? Resource! And @dusky mirage just added a Resource-like mechanism to the audio processor. So you could expose a clear, perfectly type-safe mechanism to communicate between nodes however you like.

[drowsy dome]

and then i could just string messages that way in tandem with node flow

[slate scarab]

NodeEventType isn't really designed to handle this inter-node communication.

[drowsy dome]

the issue is that of ordering

that is good though

you could make some cool fkin nodes if they could talk

i think. im trying to at least

[slate scarab]

ya you should be able to make them

[limber kernel]

did something change with custom events recently? just fetched and rebased and running into this

[dusky mirage]

Oh sorry, yes 😅

[drowsy dome]

im gonna reserve a quarter of the samples to make it happen, or open a channel and then they can talk via samples hehe

blursed idea

but still not type safe

[slate scarab]

i'll probably make a demonstration of my ideas for inter-node communication
assuming a node has access to its ID, then you could build a set of channels that, for example, send midi messages between nodes completely in user space

(with the ProcStore API)

[drowsy dome]

ok :>

https://github.com/BillyDM/Firewheel/blob/main/examples/custom_nodes/src/main.rs fwiw billy added this

[dusky mirage]

Yeah, this Resource-like thing does allow for more DAW-like features. Though I think this is as close to a DAW engine as I want to get with Firewheel. I never intended Firewheel to be a full-on DAW engine. Promising DAW-like features would make things much, much more complex in order to satisfy all of the possible use cases.

[slate scarab]

yeah and if im not mistaken, giving people this mechanism allows them to build a lot of these things themselves

which i think it really empowering

a lot of projects will need bits and pieces of more functionality, so if they have the ability to work it in within user space, then they don't need to rewrite the whole audio engine just to satisfy some niche requirement

[dusky mirage]

Though the main lacking feature that sets it apart from a true DAW engine is multi threaded processing.

[drowsy dome]

oh

rip i was mistaken i realize that example does not use the proc store

shame myself to sleep

[dusky mirage]

I of course do have an actual DAW engine I'm cooking up, but it's also not designed to be super generic like Firewheel. I couldn't imagine trying to coordinate scheduled generic events across multiple threads.

[slate scarab]

ya sounds like a pain

[dusky mirage]

Though I have learned a lot from creating Firewheel.

[limber kernel]

oh okay i just need to wrap in an option?

[dusky mirage]

It's based quite heavily around the CLAP plugin spec. https://github.com/free-audio/clap

[dusky mirage]

What are you trying to do again?

[limber kernel]

it was working prior with downcast mut on this type, im kinda not sure what changed. It looks like now I call downcast on patch instead maybe though

so

 NodeEventType::Custom(_) => {
    if let Some(impulse_response) = patch.downcast_mut::<Option<ImpulseResponse>>()
    {
...

now i think?

[dusky mirage]

Yeah, I changed it so that you can more easily downcast to the desired type directly instead of just a reference to it. Someone needed that for their node.

[limber kernel]

okay cool makes sense!

[limber kernel]

alright i think everything is pretty much good now, updated my pr @dusky mirage
a couple points: 1. i didnt include the IR in the conv node like you mentioned after chatting with Corvus as it would be a headache - makes sense I think to make some better ergonomics in seedling after this, but lmk if that is a dealbreaker. Also, 2 - I know you mentioned something about ensuring the sample rate for the IR is correct, but not sure if that is possible on the node since it only operates on the SampleResourceF32 trait 🤔

[slate scarab]

Okay, maybe this seems a little silly, but I think it's kinda brilliant. Allow me to introduce the zero input, zero output proc store communicator node.

#[derive(Component, Clone)]
struct ReverbDataNode;

impl AudioNode for ReverbDataNode {
    type Configuration = EmptyConfig;

    fn info(&self, _: &Self::Configuration) -> AudioNodeInfo {
        AudioNodeInfo::new().channel_config(ChannelConfig::new(0, 0))
    }

    fn construct_processor(
        &self,
        _configuration: &Self::Configuration,
        _cx: ConstructProcessorContext,
    ) -> impl AudioNodeProcessor {
        Self
    }
}

impl AudioNodeProcessor for ReverbDataNode {
    fn process(
        &mut self,
        _info: &ProcInfo,
        _buffers: ProcBuffers,
        events: &mut ProcEvents,
        extra: &mut ProcExtra,
    ) -> ProcessStatus {
        for mut event in events.drain() {
            if let Some(params) = event.downcast::<ReflectionEffectParams>() {
                if let Err(params) = extra.store.insert(SharedReverbData(params)) {
                    extra.store.get_mut::<SharedReverbData>().0 = params.0;
                }
            }
        }

        ProcessStatus::ClearAllOutputs
    }
}

The idea here @dusky mirage is that we want to get some reverb data into the proc store, and it changes every frame. I was thinking about how maybe we could send a closure to the audio context or something that could operate on the store. But then I thought why not just add a node that does it?

Is there any significant cost to inserting a node like this into the graph? I would also need to do this for my MIDI communication idea.

I guess technically you could run into issues with ordering.

[rapid hedge]

I started work on the steam audio library, at least the parts I can do before the review

Was this approach here merely annoying or impossible?
Sample player -> 3x some node in parallel -> decoder node

[slate scarab]

Right now, it's not expressible with sample_effects!.

[rapid hedge]

Heck

Well, single node it is then

[slate scarab]

I incorporated BillyDM's latest work locally, including the funny business above. It successfully avoids the ArcGc<OwnedGc<...>> business. Although I'd like to get his input before moving forward on that.

I should be able to review the actual PR on audionimbus now though

[rapid hedge]

I assume the hypothetical syntax would be something like

// serial
sample_effects![
(A, B, C)
]
// parallel 
sample_effects![
A, B, C
]

To mirror children!, though that would break all current user effects lol

[slate scarab]

Another issue is that you no longer get to express multiple components on each effect, which is really important!

I may simply wait until BSN lands. Expressing an arbitrary graph in a declarative way is not possible with bundles.

[rapid hedge]

Oh right

[rapid hedge]

Does BSN solve that?

[slate scarab]

See here for plans: https://github.com/bevyengine/bevy/pull/20158#discussion_r2210698776

[rapid hedge]

Oh hey neat

What happens when I connect two nodes with the same amount of output to one shared node?

Do the outputs get summed or averaged?

[slate scarab]

It's still a little difficult to conceptualize how it would work. Right now, there isn't really any ambiguity (for the most part). When you supply a couple effects to sample_effects, you're necessarily referring to one destination. That is:

commands.spawn((
    SamplePool(MyPool),
    sample_effects![
        VolumeNode::default(),
        FreeverbNode::default(),
    ],
));

commands.spawn((
    MyPool,
    SamplePlayer::new(server.load("something")),
    sample_effects![VolumeNode::from_volume(Volume::SILENT)],
));

In this case, the volume node you set in the sample player has an obvious target -- there's only one in the template! But once you can start expressing completely arbitrary graphs, not just chains, it becomes much harder to match the instantiation to the template.

[slate scarab]

just summed i believe

which is what you want imo -- mixing (with averaged gain) is a more niche use case

[slate scarab]

I guess it's actually not that hard. We could just have a GraphLabel component or something if you want to refer to a specific node.

It does become less concise to express mere chains though.

[rapid hedge]

@slate scarab re: removing firewheel from audionimbus

I guess I could send that as an event too

but it really feels like something that should be on the node

and doing the hokey pokey is ass

oh wait

that should just be a Transform anyways

well, we don't have Diff / Patch for Transform

But it could be a pos + rot

did you implement Diff / Patch for those types already? I forgor

[slate scarab]

whiiiiiich we also don't have haha (no Quat yet)

[rapid hedge]

welp

wrapper it is then

[slate scarab]

If you think CoordinateSystem is the correct thing to have on the node, then I agree that could stay. Otherwise, if the lack of Transform / Quat is a blocker this instant, then yeah some wrapper may have to do. I can submit a PR for at least Quat later today.

[rapid hedge]

The cleaner option is to use Vec3 + Quat here imo

I'll remove the firewheel feature as you suggested

@dusky mirage the downcast method on the event is exactly what I needed, thanks! Just wondering if there's a simpler way to construct an event?

[slate scarab]

NodeEventType::custom

[rapid hedge]

aha!

great 😄

also, any docs on the new shared storage thingy?

oh wait I'm on the freeverb branch

[slate scarab]

i rebased it

[rapid hedge]

so I don't have that anyways, right?

[slate scarab]

If you pull, you'll have to update bevy_seedling too.

(v0.6.0 has new commits)

[rapid hedge]

thx

hmm so all nodes share a global ProcStore?

And they can push an pull stuff into and out of it?

Is that right?

[slate scarab]

ya

[rapid hedge]

so reading it from a processor looks simple

but how do I fill it from the ECS?

[slate scarab]

The trick is getting the simulation data into it from the ECS. Without a node to do that, you can't actually get at the proc store with the context. This is why I did this.

[rapid hedge]

oh I see

I'm wondering if this is the right abstraction though?

[slate scarab]

While cute, it has one main drawback (that's obvious to me anyway); it's difficult to guarantee that this node will be processed before the others.

[rapid hedge]

The simulation outputs are per-source

so I don't really want to share them with more than one node

[slate scarab]

The reverb output is not per-source. Each one uses the listener's reflect data.

[rapid hedge]

that is true

[slate scarab]

So that's the only thing you need to share.

The per-source outputs can be sent to each effect directly.

Like they are now.

[rapid hedge]

how do you wire this up?

oh wait I can just plop it into the pool

[slate scarab]

you only need one

you can just spawn it

[rapid hedge]

gotcha

[slate scarab]

this what i did

it work
but maybe have one (audio) frame latency for reverb

[rapid hedge]

why not this?

oh wait

because the sample effects run before that

and it's not a sample effect because it only exists once

[slate scarab]

yeah even if you did that and it acted as a pass-through for the audio data, how can you guarantee that you've chosen the correct ReverbDataNode to send the single message to?

In other words, you might actually choose one that results in the rest of the pool being processed before Firewheel gets to it (and it inserts the new simulation results into the store).

If Firewheel processed all 0-input, 0-output nodes first, it would be guaranteed to work correctly. But maybe it would be better to be able to have a proper API to send data to insert into the graph from the ECS.

This would be relatively easy I think -- the FirewheelProcessor could just have a channel that receives (TypeId, Option<OwnedGc<dyn Any + Send>>).

[rapid hedge]

Oh an Entry API like on HashMap would also be nice to have on the proc store

Then you don't need this song and dance

lemme open an issue

[slate scarab]

When replacing entries that are already in the store with an API like this, the OwnedGc will guarantee that nothing is dropped improperly on the audio thread.

I should have no problem submitting a PR for this.

[dusky mirage]

Interesting. There shouldn't be any significant cost to doing this.

[dusky mirage]

I suppose it would make sense to add a get_or_insert method for the store.

[slate scarab]

ya jan also suggested an Entry API

[dusky mirage]

Ok, I added a ProcStoreEntry type!

[dusky mirage]

Interesting. That should be possible to do in theory.

Or another possibility is to just create a special type of node trait for this purpose. Which do you think would make more sense for the API?

[slate scarab]

hmmmmm

[dusky mirage]

Hmm, actually it might need to be its own process to make sure that scheduled events are processed correctly.

Because if the zero-input node gets multiple scheduled events in a block, nodes down the chain will only see the latest update.

[slate scarab]

Whatever is the simplest implementation that allows you to set data in the store before anything else would work for me haha.

oh i guess that's true

Does it "just work" if it's a node? Or does it have to actually have connections to participate in the schedule event blocks?

[dusky mirage]

What I'm thinking is that the event scheduling system will need to detect if a zero-input-output node has a new scheduled event before the end of a processing block. If it does, the engine will simply reduce the number of frames for that processing block to the number of frames before the next event.

Or we could just not bother with that, and accept that events may not be properly scheduled for these types of nodes.

Huh, in a way, these "zero input-output" nodes are kind of like uniforms in graphics processing.

Although I think it should be possible to correctly handle scheduled events. I'll work on that today.

[limber kernel]

@slate scarab im looking at your new_stream impl here for Freeverb - is this basically called whenever the sample rate changes? I'm trying to understand if I need something similar in convolution.

[slate scarab]

Purely when a new stream starts I believe, which may include changing the sample rate (so you could actually check in that method if it's changed).

There's a previous sample rate prameter in the info struct.

[limber kernel]

oh, so is that then basically just setting default sample rates for the node when it is instantiated? Actually let me just read a bit

[slate scarab]

No it'll only be called when it changes after initialization. The initial sample rates are given in the construct_processor method.

[limber kernel]

what is the use case for this? Reacting to system sample rate changes?

[slate scarab]

thats a big one

[limber kernel]

mm so nodes arent recreated after an event like that

Okay, makes sense. in my case i suppose i would just stop and reset for the convolution node, as the IRs would be at an incorrect sample rate, and theres not really a way to change on the fly

hmm sorry if this is obvious but how is this handeled for loaded samples? Are they resampled from the source asset file when the system sample rate changes (or is that even necessary?)

[slate scarab]

i wonder if it's better to persist that state anyway like simply clearing might be more jarring than a sudden shift in pitch... although iguess that's kindof effect dependent

[limber kernel]

i agree, but i also worry that the pitch shift might go unnoticed

[slate scarab]

bevy_seedling handles this situation by, when it can, jotting down where the playhead is, updating the stream, and then reloading the samples with the new sample rate and re-playing them where they left off

It'll emit a StreamRestartEvent in this scenario, so you can determine in the ECS whether to reload samples.

[limber kernel]

oh hmm i can alloc in new_stream, so ideally I could just reload the IR sample and replace

that would probably be a seedling thing, i guess. have an IR component own its asset or whatever and just update the attached convolver when the sample rate changes.

since i only have access to the SampleResource trait in the convolution node

[slate scarab]

yeah i mean there's trade offs either way

[dusky mirage]

@slate scarab Alright, I added support for "pre process" nodes! Zero testing has been done on it though. https://github.com/BillyDM/Firewheel/pull/82

[dusky mirage]

Tempted to get nitro just for those cat nodding reactions 😅

[slate scarab]

thats how they get you

then they start serving you ads

but you can't let go of your emotes

[limber kernel]

i literally did that yesterday

they make me look uncool if i can only thumbs up react to everything!!!

extortion

[rapid hedge]

It's such a mystery which Steam Audio functions are safe to call in parallel and which aren't

I'm so happy they open-sourced it

Since I can just check the Unity impl

then at least I know I'm not the only one doing it this way lol

There's one thing they do which I for sure know is writing to the same struct twice

but I guess it's not a problem in practice?????

I'll still mark it as &mut self in audionimbus though

no need to repeat their mistakes

I don't even know how to properly encode their invariants into Rust

run_direct and run_reflections can run in parallel

but you cannot run run_direct twice in parallel

not that you should, obviously

Also interestingly, they do fetch the source outputs on their audio thread

But I like our design more

[slate scarab]

I thought steam audio strongly recommended against that when you're simulating occlusion or doing other path-related stuff.

[rapid hedge]

they recommend against running the simulation on the audio thread

but fetching the outputs seems to be just reading some buffers

[slate scarab]

Hm, so the sources are shared between the game code and the audio code?

but ya i do like the direction we're headed in

[rapid hedge]

oh no sorry I misread

they only render the audio in the audio thread

they also have some kind of state store that they push and pull the outputs from

AFAICT

[slate scarab]

oh i see

[rapid hedge]

direct audio is simulated every frame

and a background thread simulates the reverb + pathing on a configurable interval

I see a disturbing lack of synchronization there

[slate scarab]

the API feels like a black box that you hope doesn't randomly explode when you look at it wrong 😅

[rapid hedge]

yeah definitely

[slate scarab]

maybe that's a little hyperbolic, but it is a nagging feeling

[rapid hedge]

with the occasional comment that goes like "oh hey do this one in a separate thread"

and I'm like

okay, but what do I need to synchronize myself in that case??? What do you do for me???

[slate scarab]

here be dragons or something

[rapid hedge]

Note for self: it looks like the only thing running in a separate thread is RunReflections and RunPathing. Reading the previous outputs and setting the new shared inputs these simulations is always done right before telling the thread to run another reflection/pathing simulation

Except when you tell Unity to use its own raytracer for Steam Audio

hehe

It's !Send

[slate scarab]

huh, wonder why

[rapid hedge]

Are all my nodes automatically rebuilt when the sampling rate changes?

[rapid hedge]

@potent moat how readily would you part with the bevy_steam_audio crate name?

Asking because I believe you have that one reserved but unused

And because I think my integration library will be done by next week 🙂

WIP: https://github.com/janhohenheim/bevy_steam_audio

(it doesn't compile yet)

If you wanna keep the name that's completely fine!

Can also call it seedling_steam_audio, seedling_audionimbus, bevy_audionimbus, bevy_nimbus, seedling_nimbus, bevy_audio_but_actually_it_is_steam_audio, etc.

[potent moat]

id part for a steep price...

1 whole rat pic

[rapid hedge]

These here tell a story

[limber kernel]

nope, it calls new_stream and keeps the node

seems like we are running into the same things in parallel haha

[limber kernel]

re this convo

[rapid hedge]

yeah totally haha

I definitely need to recreate the audio processor when the sampling rate changes

[slate scarab]

nop, there’s a method on audio node processor

oh ive been beaten

[rapid hedge]

Is there a shortcut to instead completely rebuild it?

Is it enough to trigger change detection for the config?

[slate scarab]

ya that would do it -- you could also remove and re-insert each node if you want

oh actually no the configs do need to be different

they're compared with PartialEq, if you wanted to do it that way

[rapid hedge]

Alright, that’s doable. Can just add the sample rate to them as a field

[slate scarab]

Mm you shouldn't need a workaround like that -- imo reinsertion of the actual node component is better.

[rapid hedge]

Also, API wise, I currently have two resources for global settings. One is for "you can change these quality settings every frame is you want" and one is "changing these quality settings will completely rebuild everything"

I think that’s nice

But naming them is hard haha

[rapid hedge]

Hmm I don’t which. The ones in the pool? Will the audio graph nodes then also get reinserted?

Or I guess I need to reinsert the effects nodes per sampler

[slate scarab]

Ah, sorry, actually this won't trigger the behavior. It's a little tricky since you want to restore the connections. So messing with the config is the only way to get this behavior going because I specifically wrote logic to handle re-connecting everything.

[rapid hedge]

More stuff for the docs 😉

[slate scarab]

I can generalize that though -- a "recreate" event could be provided, and updating the config could trigger it.

[rapid hedge]

Ah, so I could just call that recreate event manually then?

[slate scarab]

right

[rapid hedge]

That would be neat

Hmm

I just though of something

Do I yeet the remaining stuff in the input and out buffer?

[slate scarab]

It would be very natural if the connections were persisted in the ECS (but that's blocked due to no many-to-many). So you specifically have to go searching in the audio context and match up the edges.

[rapid hedge]

Remember we accumulate those

[slate scarab]

imo it's better to not perturb that -- you'll get a subtle shift in frequency after a sample rate change, but I think that's better than a full dropout

[rapid hedge]

Oh, in that case I can’t do a full rebuild

[slate scarab]

Are you forced to rebuild because you need the steam audio context?

[rapid hedge]

It’s because changing the sampling rate forces me to create a new simulation

And I think the node processors contain some audionimbus types derived from the previous simulation

Specifically, their constructors ask for the sampling rate

[slate scarab]

well, you get the new sampling rate in the StreamInfo struct https://docs.rs/firewheel/latest/firewheel/struct.StreamInfo.html

[rapid hedge]

Yeah I just thought I wanted to avoid duplicating the creation code across two functions

Because if I add a new field, I have to remember to also update it there

But yeah it’s fine

[slate scarab]

you should be able to create a function that you call in both places i'd think and using the new_stream method is more efficient and allows you to (potentially) persist state

Looks like the API doesn't make it easy to change sample rates on most types without rebuilding though so

but at least you wouldn't have to clear the fixed buffers

[rapid hedge]

Another thing: when two sampler players belong to the same pool, can I use separate sample effect configurations for them?

[slate scarab]

They have to have the same shape (that's part of the idea behind the pools). While allowing different configurations would be neat, it would potentially require constant re-routing as you play new samples, which incurs audio graph recompilation (which isn't ideal if it's happening all the time).

[rapid hedge]

So if I want the user to be able to tweak the params per source, I cannot use pools?

[slate scarab]

It depends on what you mean by "tweak the params." Do you mean just changing their values?

[rapid hedge]

Adjust the params per sample player

(On spawn ideally)

[slate scarab]

Yes, you can do that. It would be quite limiting otherwise! You just can't add new effects or re-arrange them.

[rapid hedge]

Ah okay, that’s good 🙂

[slate scarab]

(If you give them in the wrong order, they're automatically corrected, to be clear.)

[rapid hedge]

How? Do I just spawn the same hierarchy as in the pool?

[slate scarab]

#[derive(PoolLabel, Debug, Clone, PartialEq, Eq, Hash)]
struct SpatialPool;

commands.spawn((
    SamplerPool(SpatialPool),
    sample_effects![
      SpatialBasicNode::default(),
      VolumeNode::default(),
    ],
));

// this works
commands.spawn((SpatialPool, SamplePlayer::new(server.load("my_sample.wav"))));

// and this
commands.spawn((
    SpatialPool, 
    SamplePlayer::new(server.load("my_sample.wav")),
    sample_effects![VolumeNode::default()],
));

// even this (the effects are re-ordered after insertion)
commands.spawn((
    SpatialPool, 
    SamplePlayer::new(server.load("my_sample.wav")),
    sample_effects![
        VolumeNode::default(),
        SpatialBasicNode::default(),
    ],
));

// but you can't do this (the stray effect is removed with a warning)
commands.spawn((
    SpatialPool, 
    SamplePlayer::new(server.load("my_sample.wav")),
    sample_effects![
        VolumeNode::default(),
        SpatialBasicNode::default(),
        LowPassNode::default(),
    ],
));

(docs here: https://docs.rs/bevy_seedling/latest/bevy_seedling/pool/sample_effects/struct.SampleEffects.html#static-pools)

If you're feeling spicy, you can throw the pools to the wind and just spawn whatever you want. If the effects associated with a SamplePlayer don't match the default pool, and the entity doesn't otherwise have a label, a new pool is dynamically created for it.

(Or if such a dynamic pool already exists, it'll be queued in that.)

see: https://docs.rs/bevy_seedling/latest/bevy_seedling/pool/dynamic/index.html

This might be over the line of "too much magic." But I think it's neat.

[rapid hedge]

It’s really neat, thanks!

How much of a guarantee do I have over when StreamEventStart is sent?

Or StreamStartEvent

The event suffix should be gone >:[

[slate scarab]

StreamStartEvent will always be fired once in PostUpdate assuming stream initialization is successful.

[rapid hedge]

hmm

So users setting up their scenes in Update or even Startup cannot know the sample rate yet

[slate scarab]

Yes, it's a balance between allowing stream configuration before initialization and facilitating other setup.

[rapid hedge]

So when they set up nodes, they must either

• delay their setup or
• spawn nodes will their config full of Options for late init

[slate scarab]

There's also a system set for this initializaation, so you can order normal systems after it.

[rapid hedge]

Tending towards the latter so this detail is hidden from the users

[slate scarab]

spawn nodes will their config full of Options for late init
but why would the config need anything like sample rate?

[rapid hedge]

Because to create the processor you need the sample rate

[slate scarab]

Yeah but that's given to them when they're created. It should not be placed in a config struct.

[rapid hedge]

Oh, they’re not created right away?

Riiiiiiight

[slate scarab]

No, always deferred. So you can totally spawn stuff whenever, assuming it doesn't need direct access to the AudioContext or any information derived from it.

[rapid hedge]

Let’s see, do I need the simulator to create the processors? I hope not

[slate scarab]

That's just for the sources, right? But you don't actually need the sources to create the effects I think.

[rapid hedge]

Indeed I do not! Yay!!!

What happens in this scenario?

// In the steam audio library
commands
    .spawn((
        SamplerPool(SteamAudioPool),
        VolumeNode::default(),
        VolumeNodeConfig {
            channels: NonZeroChannelCount::new(quality.num_channels()).unwrap(),
        },
        sample_effects![ambisonic_node],
    ))
    // we only need one decoder
    .chain_node(ambisonic_decode_node);

And then the user spawns this

commands
    .spawn((
        SteamAudioPool,
        SamplePlayer::new(...),
        sample_effects![AmbisonicNode::default()],
    ))

will it also have the VolumeNodeConfig? And will it also chain into the one global decoder?

I suppose that global decoder entity also needs to be available for the users if they want to set up their own pools

[slate scarab]

No, only the stuff within sample_effects will be copied around, so the volume config won't exist on the sample player. All the pool nodes are always routed to the pool entity itself, and in this case there's a volume node there. So anything you connect the pool entity to will necessarily route all of the samplers in that pool through it.

[rapid hedge]

Ah, so I could add the volume node config on an On Add observer for the pool, right?

[slate scarab]

ya

since it won't read that until Last

[rapid hedge]

the volume node is always there, right?

[slate scarab]

so you can sneak it in there

[slate scarab]

By default, yes. But you can use a different node and it won't be overwritten.

[rapid hedge]

So is this sensible as an end-user API?

It certainly looks neato

[slate scarab]

ya it seems good

[rapid hedge]

what happens if you add additional effects to that, so that the shape of the poll doesn't match?

[slate scarab]

warning, and removed

[rapid hedge]

it just gets placed in a dynamic pool then?

[slate scarab]

(the offending effects are removed)

[rapid hedge]

aaaah

okay

Asking because I want to prevent the user from accidentally removing their chain into the decoder

So when they set up extra effects, they need to manually chain into the decoder

that's fine I think

Just need to make it accessible 🙂

[slate scarab]

Yeah you could just make a global decoder with a node label

AmbisonicDecoderBus

or something

[rapid hedge]

Oooh I didn't even consider that

I thought of letting them to Single<Entity, With<...>>

but this is way better!

[slate scarab]

conveniently, the node label allows you to also do this if you want

in addition to just using the label to connect

[rapid hedge]

Okay, then we can get some really neat API!!

[rapid hedge]

This is for spawning a sample player

the global initialization is just adding the plugin

there's a completely optional SteamAudioQuality resource for changing the global simulator settings

but each sample player can also have their own defaults overwritten by adding a config

I set up the simulator to run automatically for you on a clock, so you don't need to worry about that

[slate scarab]

oh nice

[rapid hedge]

The only thing that is not automagical is the scene creation

since you need to get a trimesh representation of the whole thing

[slate scarab]

oh right that bit

[rapid hedge]

I don't want to do that on a clock since you don't need to run it often

heck, a static scene only needs it exactly once

so I'll leave that up to the user, I think

[slate scarab]

Yeah I was hoping there would be some nice solution to just sorta derive it from physics objects or something

[rapid hedge]

over an event

[rapid hedge]

yeah totally!

I can just copy the same setup as for rerecast 🙂

There you can register a backend that creates the scene

the builtin just does it from Mesh3d

but you can depend on an extra crate that instead uses avian colliders

(wish we had first-party colliders, grumble grumble)

@sturdy prawn give first-party colliders in 0.19

thx

[slate scarab]

yes pls

[rapid hedge]

The only ugly-ish part of the story is that the user needs to manually trigger UpdateSteamAudioScene

but eh, they'll manage

[rapid hedge]

then just like in rerecast, I'll do that in a separate thread since you can have a biig scene

[rapid hedge]

It's my right as a GitHub Sponsor to just demand work, right????

[sturdy prawn]

kk will do, ez

[sturdy prawn]

I believe you were actually my first sponsor too haha

[rapid hedge]

hehehe

see, this is my loyalty bonus

I filled out my Jondolf stamp card

now I get upstream collider structs

[slate scarab]

prolly want that anyway for level changes and stuff

[rapid hedge]

yeah but it would be neat if we did something like regenerate it when static / kinematic colliders were mutated

so it's completely invisible to the user

but doing that is a little bit of a rabbit hole

[slate scarab]

ya it's probably quite expensive unless you can find ways to only rebuild isolated parts

say if you reconfigure small parts of a large level

[rapid hedge]

which you can do with steam audio

But eh, I'll leave that as a followup

rebuild everything is good for now

[slate scarab]

ya

[rapid hedge]

Is this correct?

pub(crate) fn setup_audionimbus(mut commands: Commands, quality: Res<SteamAudioQuality>) {
    commands
        .spawn((
            SamplerPool(SteamAudioPool),
            VolumeNode::default(),
            VolumeNodeConfig {
                channels: NonZeroChannelCount::new(quality.num_channels()).unwrap(),
            },
            sample_effects![(
                AudionimbusNode::default(),
                AudionimbusNodeConfig {
                    order: quality.order
                }
            )],
        ))
        // we only need one decoder
        .chain_node((
            SteamAudioDecodeBus,
            AmbisonicDecodeNode::default(),
            AmbisonicDecodeNodeConfig {
                order: quality.order,
            },
        ));
}

[slate scarab]

ya that looks correct

[rapid hedge]

I think you said leaving the frame size hardcoded is fine, right?

[slate scarab]

I was thinking about that a little more. You only need the fixed size to be larger than the max frame size if you can't be sure smaller sizes are not multipled of the fixed size. In other words, it's fine if the block size switches from 1024 to 512 if the fixed size is 256. In both cases, the fixed sizes fits perfectly within both.

Anyway, yeah fixed is probably fine.

You could make it part of a config struct or something.

A resource.

[rapid hedge]

I wonder why the user would want to touch it though

[slate scarab]

Careful control over latency.

[rapid hedge]

fair enough

I have a resource that contains stuff that all require a full rebuild

could plop that one in there

now the boring part

creating 100000 buffers in the node processor

because they are all hardcoded to the ambisonic order

heck yeah, time for audio_buffer_ptrs_1 through 10

[slate scarab]

You could remove most of the buffers if you copy each effects outputs to the output buffer immediately after they're processed.

Then you only need one buffer with nine channels (or however many).

[rapid hedge]

fuck

can I just newtype this?

should be fine, right?

[slate scarab]

well you could use billy's crate for this

[rapid hedge]

orly?

[slate scarab]

actually idk if it'll fix the sendness on its own

conceptually it should though
oh but no it's specifically for references

but you could use this idea

for pointers

and then insist to the compiler that the type is Send and Sync

[rapid hedge]

Oh neat, I see

[slate scarab]

Since it'll never hold anything except for when you're using it.

[rapid hedge]

I'll do this for now to get it to work at all

[slate scarab]

at which point the mutable reference guarantees nothing else is looking at it

[rapid hedge]

but I'll keep it in mind 🙂

[slate scarab]

in fact i think the crate could be generalized to anything -- you could insist 'static + Send + Sync regardless of its contents with this API

it doesn't need special case handling for references i think..... (maybe?)

[rapid hedge]

Could the config struct maybe be relaxed to require FromWorld instead of Default?

[slate scarab]

oh how does that interact with Default? Do all Default types implement FromWorld?

(yes)

[rapid hedge]

yep 🙂

[slate scarab]

is that sufficient for required components?

[rapid hedge]

Uuuuh lemme check

nope

[slate scarab]

aw man

[rapid hedge]

huh, wonder why

let's ask ecs dev

[slate scarab]

can't even do register_required_components_with

[slate scarab]

We could do required-components-at-home to satisfy FromWorld.

Maybe, although I'm still not sure if that interacts poorly with bundles.

[rapid hedge]

Well, I have this now:

#[derive(Diff, Patch, Debug, Clone, RealtimeClone, PartialEq, Component, Default, Reflect)]
#[reflect(Component)]
#[component(on_add = on_add_audionimbus_node_config)]
pub struct AudionimbusNodeConfig {
    pub(crate) order: u32,
    pub(crate) frame_size: u32,
}

fn on_add_audionimbus_node_config(mut world: DeferredWorld, ctx: HookContext) {
    let quality = *world.resource::<SteamAudioQuality>();
    let mut entity = world.entity_mut(ctx.entity);
    let mut config = entity.get_mut::<AudionimbusNodeConfig>().unwrap();
    config.order = quality.order;
    config.frame_size = quality.frame_size;
}

that way I have a bogus default that instantly gets replaced

Bonus: this will panic when the node is spawned with Disable

[slate scarab]

query observers when

fn inject_config(
    config: Start<Entity, With<AudioNimbusNodeConfig>>, 
    quality: Res<SteamAudioQuality>,
    mut commands: Commands,
) {
    // ...
}

Same semantics, but correctly handles Disabled. I have a crate that does this rn :3

[rapid hedge]

Is it triggered before all On Add observers?

[slate scarab]

No, it would run in an Add observer. But that's probably fine in this case?

[rapid hedge]

hmm not quite semantically correct

AFAIK these kinds of invariants should be in a hook

since the component is fundamentally in a corrupt state otherwise

now how do I add a filter to a hook

Or I guess .get_mut() doesn't care about default filters?

eh, idk

[slate scarab]

i don't think it does

[rapid hedge]

it will be alright

you use disabled, you submit a PR

[slate scarab]

anyway i got excited because Disabled
if the ecosystem used this approach instead of normal observers, Disabled would actually be usable

(the hook is better in this case but maybe you get the idea)

[rapid hedge]

yep

see also https://github.com/TheBevyFlock/bevy_cli/issues/524

When I go and recreate the node configs because the global quality settings changed, which config instances do I mutate?

The ones on the sample players?

[slate scarab]

The ones in the pool, yes. All the effects are children of the pool, so you should be able to iter descendents and find all Config, With<Node>

probably the best way alternatively, just query for all Config, Without<EffectOf>

(the live ones will never have EffectOf)

[rapid hedge]

hold up

I thought it was also spawned for each sample player

so

• one related to sample player
• one on the pool
• one in the graph

[slate scarab]

They are, but each effect is a different entity, and the real ones are not related via EffectOf.

[rapid hedge]

but if I mutate the ones that are not the EffectOf entities, those will be out of sync, no?

[slate scarab]

Mm, I suppose newly spawned ones would receive the old config in this scenario. So yeah, probably best to iter_descendents for both Children and SampleEffects specifically on the pool entities.

That would be a surgical update, anyway. It probably really doesn't matter.

Updating all of them is easier and should produce the same effect.

[rapid hedge]

okay, I just wasn't sure if that would trigger the propagation since they mutated

which would be silly

[slate scarab]

Sorry for the circuitous suggestions.

[rapid hedge]

Like, if I mutate all FooConfigs, wouldn't that trigger propagation to update the live ones again?

Though it's only a few f32

so I don't really care much

[slate scarab]

Mutating any configs in SampleEffects won't produce any immediate effects, so if they exist in there it doesn't matter if you update them. And updating them in the pool's template is necessary to ensure any newly spawned effects (like when the pool grows) receive the new config.

so ya
just mutate the lot

[rapid hedge]

And you said mutating configs will only trigger a node rebuild if they're not PartialEq with the old value, right?

[slate scarab]

yes

[rapid hedge]

So even if I accidentally triggered the "plz rebuild the node" branch multiple times, it wouldn't actually rebuild it more than once

that's good!

Oh, also

pub(crate) fn setup_audionimbus(mut commands: Commands, quality: Res<SteamAudioQuality>) {
    commands
        .spawn((
            SamplerPool(SteamAudioPool),
            VolumeNode::default(),
            VolumeNodeConfig {
                channels: NonZeroChannelCount::new(quality.num_channels()).unwrap(),
            },
            sample_effects![(AudionimbusNode::default(),)],
        ))
        // we only need one decoder
        .chain_node((SteamAudioDecodeBus, AmbisonicDecodeNode::default()));
}

The VolumeNode and VolumeNodeConfig do literally nothing here, right?

I just asked this a while ago, I know

But uuuh I forgor

[slate scarab]

Well you don't have to mention VolumeNode if you don't want -- that's the default node in that position.

But the config is critical.

You'll want it to match the number of channels that the ambisonic nodes have.

[rapid hedge]

Hmm but you said that doesn't get copied to the actual sample players, so we need to sneak it back in with an observer IIRC

[slate scarab]

There's only one volume node.

it would be easy to communicate again with some visual node graphs haha

[rapid hedge]

[slate scarab]

Within the pool, each SamplerNode is like a slot. SamplePlayers try to claim these slots, but they're not nodes in their own right. Spawning a SamplePlayer will never re-route anything (unless the pool has to grow). So eachSamplerNode, along with its effects, has a static, unchanging connection to the node that's inserted into the SamplerPool<T> entity.

[rapid hedge]

So what was the bit with having to insert the config on an observer?

[rapid hedge]

^

[slate scarab]

I think you wanted that so users wouldn't have to specifically provide the number of channels themselves. It would just "automagically" happen later.

In other words, with an observer set up, a user could provide their own setup like this:

fn manual_setup(mut commands: Commands) {
    commands
        .spawn((
            SamplerPool(SteamAudioPool),
            sample_effects![SomeEffect::default(), (AudionimbusNode::default(),)],
        ))
        .connect(SteamAudioDecodeBus);
}

and it could "just work"

But you don't need to do that if you want to maintain full control over spawning the pool.

Probably better to have a fully automatic setup, with some docs on how the user can spawn it themselves if they want.

So I don't think the observer would be necessary.

[rapid hedge]

I don't really get why this doesn't just "use" the volume node + config from the pool

[slate scarab]

Well this is the pool definition itself.

[rapid hedge]

Oooooooooooh

okay, then we were talking past each other

probably my bad for mixing stuff up 😄

[slate scarab]

Oh, yeah SamplerNodes will not need this ever.

[rapid hedge]

I was thinking the observer would be for the place where the sample player is spawned!!

okay, good!

[slate scarab]

oop

i meant SamplePlayer

[rapid hedge]

Agreed that we don't need an observer then!

pub(crate) fn setup_audionimbus(mut commands: Commands, quality: Res<SteamAudioQuality>) {
    // we only need one decoder
    commands.spawn((SteamAudioDecodeBus, AmbisonicDecodeNode::default()));

    // Copy-paste this part if you want to set up your own pool!
    commands
        .spawn((
            SamplerPool(SteamAudioPool),
            VolumeNodeConfig {
                channels: NonZeroChannelCount::new(quality.num_channels()).unwrap(),
            },
            sample_effects![AudionimbusNode::default()],
        ))
        .chain_node(SteamAudioDecodeBus);
}

what do you think about this?

that should show users exactly what to do

Also, should this normalize?

[slate scarab]

ya that seems good :)

[slate scarab]

well you should probably be able to choose the blending parameters

[rapid hedge]

yeah I was moreso asking if it should sum or average

I assume "blend" == "weighted sum divided by sum of weights"?

[slate scarab]

I suppose "mix" would be the more accurate word here. I'm not sure exactly what the standard procedure is for mixing spatialized elements like reflection and reverb is. Usually you wouldn't necessarily normalize them though.

[rapid hedge]

yeah that's what I thought

Seems to me like the normalization would be counter-intuitive?

like, if I up the reflection gain, I wouldn't expect the direct gain to go down

[slate scarab]

Yeah usually these effects are "additions" to the direct sound.

But that might be debatable.

It's just an approximation anyway, so it's not like it's "physically accurate" either way.

I don't think the difference really matters though -- you could achieve the same effect as a user by adjusting the source's volume.

[rapid hedge]

I'll check unity

[rapid hedge]

Hmm

I wonder how the heck I would even do a generic scene backend

For avian, I can make some assumptions. One collider = one acoustic material, static rigid bodies don't change their position, dynamic / kinematic bodies do

I guess for the nonavian case it could be one Mesh3d = one material, and you need to mark which objects are static?

And you need to add components for all acoustic materials ofc

it's just 3 floats, so in theory no need to fiddle with handles

[slate scarab]

oh ya, but that seems pretty easy

[rapid hedge]

But in practice, steam audio treats them as handles

[rapid hedge]

It gets more difficult when you notice that steam audio also keeps handles to its internal representation of the meshes

so I need to make sure that when a mesh is despawned in Bevy, it's despawned in steam audio

but wait, in avian, it should instead be "when a collider is despawned", not "when a mesh is despawned"

But also, there shouldn't be one mesh per avian collider instance. An Avian collider is a handle in a trenchcoat, it uses an Arc under the hood

so it should be one steam audio mesh per avian collider arc

So the steam audio mesh needs to be despawned once all avian colliders sharing the same arc are despawned

you see how it becomes more difficult to generalize this?

Guess the best way is to not do too much out of the box and let each scene backend do its own thing

[slate scarab]

but wouldn't the backend mean you don't need to generalize

[rapid hedge]

yeah it's just a bunch of stuff to book keep

And I don't like that multiple backends need to have very similar code and a lot of boilerplate

Hmm @sturdy prawn this is now the second crate that needs a generic "plz turn this collider into a trimesh, thx" method

can I convince you to upstream it now? 😛 (into Avian)

[rapid hedge]

update: unity too just adds them

[rapid hedge]

Okay, I think I have a clean idea of how the scene can be built from meshes so that it Just Works for people who don't want to use avian

I can use the AssetEvent<Mesh> messages to synchronize the steam audio mesh handles with Bevy's own mesh handles

then a Mesh3d + Transform corresponds to what steam audio calls an "instanced mesh", which is a subscene containing static meshes that has a transform applied

Then I should be able to check if the Transform of a Mesh3d changed this update, and if yes, move the instanced mesh according to the GlobalTransformon the Bevy side

All of that in PostUpdate after the transform propagation

and since seedling runs in Last, it should have up to date data for everything

Then for avian I have to do some funky DIY shit to pretend that colliders are handles (they should be handles if we didn't use parry)

but then the same approach would work

Just replace "did transform change" with "is collider not asleep"

this approach also means that users don't need to care about updating their steam audio scene

it Just Works

[rapid hedge]

On this line

@slate scarab any idea?

OOOOH It should be .connect

[slate scarab]

oh yes -- chain_node is for spawning new nodes :3 but that could be quite confusing since the label is a component 😅

i wonder if we should warn if more than one entities with a given label exist

That restricts the utility of the labels a tiny bit, although it's probably a bad idea to insert them on anything but an audio node anyway.

[rapid hedge]

What do you think of this API?

fn main() {
    App::new()
        .add_plugins((
            DefaultPlugins,
            SeedlingPlugin::default(),
            // Add the SteamAudioPlugin to the app to enable Steam Audio functionality
            SteamAudioPlugin::default(),
            // Steam Audio still needs some scene backend to know how to build its 3D scene.
            // Mesh3dBackendPlugin does this by using all entities that hold both
            // `Mesh3d` and `MeshMaterial3d`.
            Mesh3dBackendPlugin::default(),
        ))
        .add_systems(Startup, setup)
        .run();
}

fn setup(
    mut commands: Commands,
    assets: Res<AssetServer>,
    mut meshes: ResMut<Assets<Mesh>>,
    mut materials: ResMut<Assets<StandardMaterial>>,
) {
    // The camera is our listener using  SteamAudioListener
    commands.spawn((Camera3d::default(), SteamAudioListener));

    // Some occluding geometry using MeshSteamAudioMaterial
    commands.spawn((
        Mesh3d(meshes.add(Cuboid::new(3.0, 2.0, 0.5))),
        MeshMaterial3d(materials.add(Color::WHITE)),
        Transform::from_xyz(0.0, 0.0, -4.0),
        MeshSteamAudioMaterial(SteamAudioMaterial::GENERIC),
    ));

    // The sample player uses Steam Audio through the SteamAudioPool
    commands.spawn((
        SamplePlayer::new(assets.load("selfless_courage.ogg")),
        SteamAudioPool,
        Transform::from_xyz(6.0, 0.0, 0.0),
        Mesh3d(meshes.add(Sphere::new(0.5))),
        MeshMaterial3d(materials.add(Color::WHITE)),
    ));

    commands.spawn((
        DirectionalLight::default(),
        Transform::default().looking_to(Vec3::new(0.5, -1.0, -0.3), Vec3::Y),
    ));
}

This is the current miminal example 🙂

[rapid hedge]

Also ping @cold isle, you'll like this

[cold isle]

I like what I'm seeing a lot 🙂 does it automatically sync Bevy's transforms with SA?

Thanks again @rapid hedge for the PR and @slate scarab for the review, really appreciate it!

[rapid hedge]

yep!

[rapid hedge]

thanks for merging!

Btw, did you know that running run_reflections without setting a scene first gives a segfault? 😄

[cold isle]

Huhhhh

no? 😄

With the current version? I pushed a fix a few versions ago that was supposed to prevent this

Ha actually it wasn't about not setting the scene, it was when passing incorrect settings. Hmm interesting

[cold isle]

It looks great 🙂

[rapid hedge]

do you have time to help me debug another segfault?

[rapid hedge]

Welp, I fixed it by upping max_num_rays, somehow
edit: bleh, my convolutions had a higher num_rays than my global maximum. There's no graceful handler for that, it seems.

it works now!

@slate scarab mind taking a look? 🙂
https://github.com/janhohenheim/bevy_steam_audio

[rapid hedge]

Hmm I have my per-source config currently split between the node, node config, and a non-seedling config component, depending on its effect on firewheel

node = this has stuff that firewheel uses
node config = this means we need to realloc stuff in firewheel
extra config = this is just used by the simulation that runs in the ECS

which makes sense for me as the crate author

but I think this will be lost on users

"Where is the FooSetting? Is it in the node or that weird second settings component?"

Do you think it's fine to add settings that are not used by firewheel to the node?

so that the ECS can query that instead of a dedicated non-firewheel settings component

Or is that a semantic no-go?

[rapid hedge]

Also, @potent moat mind giving me an owner invite for https://crates.io/crates/bevy-steam-audio ? If @cold isle also publishes a release soon, I can also release an alpha 🙂

[slate scarab]

hm, what kinds of settings are those in this case?

i suppose ill find out after i take a look!

[rapid hedge]

The ones used in set_inputs

Which are hardcoded rn

[slate scarab]

ya i think it’s fine to put those in the node, you can always ignore them for diffing

[rapid hedge]

Btw the output buffers are allocating in the custom config example, mind checking that out?

[rapid hedge]

Cool!

[slate scarab]

i didn’t even know we had a check for that 😅

[rapid hedge]

And the pathing stuff is still WIP, so it has no effect yet 🙂

[rapid hedge]

I added one!

[slate scarab]

oh i see

[rapid hedge]

Maybe the check is faulty though

[rapid hedge]

Hmm, it would be better if I just compared the capacity at the beginning and end of the process

Then I could just do start_cap = a.capacity() + b.capacity() * …

Since capacities only grow, any change in that number must be an allocation

I'll do that real quick

[slate scarab]

(you can shrink to fit )

[rapid hedge]

yeah but I would have to manually call that

so that's not something that can accidentally happen through a bug I believe

@slate scarab try pulling again 🙂

[slate scarab]

haha i love the minimal example

[rapid hedge]

it speeeens

[potent moat]

Ya will try to do so later tonight

[rapid hedge]

thx!

no rush though 🙂

[slate scarab]

@rapid hedge I've been doing a bit of work to help abstract the fixed buffering and reduce the amount of copying in general. I've applied this to the decode node already, and I'll get to the encoder next. Once that's done (tomorrow sometime) I'll submit a PR if you don't mind :)

[rapid hedge]

heck yeah!!!

Let me just give you collab rights, sec

Click plz https://github.com/janhohenheim/bevy_steam_audio/invitations

1 entity?????
Is GitHub == ECS??????

[slate scarab]

lmao

[rapid hedge]

I've in the meantime tried to add pathing support

but I'm getting SEGFAULT instead lol

There's an integration test for pathing in audionimbus

but uuuuh

It's not wired up properly

so the outputs are always just 0.0

and the test only checks if it panics or not

[slate scarab]

mfw segfault: 0.0

[rapid hedge]

there's a ton of SEGFAULTSs that you can reach with audionimbus

But tbh I wouldn't have been able to do it better

the safety mechanism of steam audio really don't map well onto Rust

it's like