Better Audio

[slate scarab]

The dynamic pool logic has been moved to Last, actually.

The context probably isn’t restarting (that would only happen in response to, say, changing output devices).

And send connections are just queued like any other connection, so the effect they produce shouldn’t result in any kind of race

I’d think the main snag would be when exactly the contents of the simulation results become invalid (like when removing a source). Assuming the segfault is happening in the audio processor that is

[rapid hedge]

I just wanted to tell you that I narrowed the issue down to removing the source 😄

I first thought that couldn't be it since I spawn them looping in one crashing test

but seedling does despawn those--When an underrun happens

[rapid hedge]

(or something like that)

[slate scarab]

ya it should happen when the pool runs out of space

[rapid hedge]

👍

When I comment out simulator.remove_source(&source) nothing crashes

which is of course a memory leak

Maybe I can remove them on preupdate?

nope, crashes too

[slate scarab]

Right so assuming I'm not way off base, it seems quite likely that the steam audio processors occasionally hold onto the simulation outputs from the old (removed) source, and something in there is just deallocated.

reduced to atoms

[rapid hedge]

thanos_snap.mp4

[slate scarab]

Even if we sent a message immediately to the audio thread that the data is old, they might not receive it before using the old data.

We could protect the data with a mutex or do some checks with an atomic flag, but that certainly doesn't enforce memory safety in a very rigorous way.

[rapid hedge]

But we could delay the removal until the next time the simulator is "resting"

we have an atomic flag for that

[slate scarab]

Well what I mean is that audionimbus probably couldn't guarantee memory safety if removing a source like this deallocates resources that may be shared between threads.

Without some additional work.

[rapid hedge]

the docs suggest that stuff like this should be safe if you don't call .commit(), but apparently not

Hmm though I'm not so sure

because I did try replacing all RwLock accesses with .write()

that would in effect destroy any parallelism

but it didn't help

[rapid hedge]

nope that didn't help either

okay let's try something stupid

what if I delay every source despawn by 1 second?

[rapid hedge]

that fixes it

oh nvm

it crashed after a long while

[slate scarab]

where is the segfault? have you checked with ur fancy ide

[rapid hedge]

same place as the one you ran into

Okay, delaying by 1 sec and making sure the simulator is not currently running seems to do the trick

tuning the delay down to 0.1 sec crashes again

brilliant stuff

wonder if the delay needs to be as long as the sample

yep looks like it

oh nvm it still crashed

(using 250 ms for CAW which is around 210 I believe?)

[slate scarab]

yeah i wouldn't expect those to be correlated

[rapid hedge]

it just seems like the longer it's retained, the less likely it crashes

Yeah there's no magic setting that never crashes, it seems

at least not for all files

1 sec seems to be good for CAW

but not for step1.ogg

is this a good sign?

[slate scarab]

That is currently what I expect is the issue, though I could be off-base. Anyway, the only spooky action at a distance I can think of is that the IR data is removed or set to null when the source is removed from the simulation. And the steam audio firewheel nodes have a reference to that, even after the source that produced these outputs has been removed from the simulation.

[rapid hedge]

if this is to believed, then params.fftIR->readBuffer is null

[slate scarab]

In a kinda racy way.

[rapid hedge]

In here I mean ^

Full line is ArrayMath::multiplyAccumulate(static_cast<int>(mFFTDryBlocks.size(1)), mFFTDryBlocks[index], (*params.fftIR->readBuffer)[i][j], mFFTWet[i]);

Not sure if any of that helps

[rapid hedge]

ideas for a fix?

or a way to check?

[slate scarab]

oh ya so i think mine was just an overrun of that, but it sounds like your issue is that it's straight up null

[rapid hedge]

It could also be null due to LLDB shenanigans in the current stack frame

[slate scarab]

hm i see

[rapid hedge]

though that would certainly explain the segfault

[slate scarab]

If my diagnoses isn't completely wrong, then you could use a garbage collection scheme for the sources. When all users drop the source (or the simulation outputs that depend on it), then it would be safe to remove them from the simulation.

But the fact that this can happen indicates that the audionimbus API doesn't respect Rust's type system as-is, so I don't think bevy_steam_audio should do this (at least long term).

[rapid hedge]

Figured out how to print stuff behind a unique_ptr

there's certainly a null there

this thingy is supposed to be a matrix, I believe

Idk if it's row or col major, docs say "Multidimensional array that can be accessed using standard multidimensional array notation,"

but see the sizes?

it's a 5 x 0 matrix

[slate scarab]

actually the problem might just be dropping the source while outputs are still in use

(as in unrelated to removing from the simulation)

unless the sources are re-created each simulation period

[rapid hedge]

Unity straight up drops them on OnDisable

which I believe should be the same as our On<Remove>

[slate scarab]

This is in c sharp, right? How does that interact with the underlying audio processors, though? Do they also have a copy of the source?

[rapid hedge]

nope, their audio processors are in C++

and they tell the steam API remotely to please release the handle

[rapid hedge]

they do have a copy it seems, yes 👀

the stuff is reference counted

so if the processor has a handle

it won't be dropped!

[slate scarab]

Yeah, that's what I was getting at.

[rapid hedge]

They also don't shuffle the output around. Instead, they fetch the output during the processing

[slate scarab]

Yeah we could just do that, especially if the source is just reference counted anyway.

This soundness hole still exists, but it'll be much easier to avoid.

[rapid hedge]

[slate scarab]

I think that should be quite easy to do as well, luckily

[slate scarab]

I think it would actually be pretty easy to fix from audionimbus's side too. If the sensitive simulation outputs hold a reference to the Source they were derived from, then it can't be dropped (from within Rust) in a way that would result in use-after-free.

[rapid hedge]

@slate scarab EffectOf is only on the nodes spawned directly by the user with sample_effects![], right?

[slate scarab]

Well, not strictly -- they may be inserted after a SamplePlayer is spawned in a pool and doesn't specify all the effects itself.

[rapid hedge]

How can I get the SamplePlayer of a new SteamAudioNode effect then?

I want to bind the lifetime of one shared copy of the SA Source to it

(if that makes sense)

[slate scarab]

Hm, I'm not sure you'd need to do much at all, really, assuming the SteamAudioNode will hold the source itself.

If all you want to do is remove a source after it's no longer in use, you could watch for On<Replace, FollowerOf> and filter down to just entities with a SteamAudioNode. On this event, send a message to the node that its source is to be dropped. If dropping involved deallocation, then it should hold the source as an ArcGc<Source>.

Let me know if that makes sense.

[rapid hedge]

that does make sense

I don't care much about the source that is in the processor

yeah I didn't consider that the node should hold it

I'll do that

or not

[slate scarab]

It shouldn't be a part of the node imo. Just send it to the processor like the simulation outputs.

[rapid hedge]

that does make sense

[slate scarab]

Sorry, I sometimes use the vocabulary a little loosely.

Where "node" can mean the handle or the actual processor in the graph.

even though we call the handles FooNode

I think initially we called them FooParams, but I somewhat liked conceiving of the params as the nodes for users who don't dig into the internals.

[rapid hedge]

oh, I made a tiny mistake in my explanation above. They actually do read the outputs same as we do, they just store it in a wrapper that looks like the steam audio source

but already has the outputs read out

[slate scarab]

Huh...

[rapid hedge]

and yeah, holds a shared pointer

[slate scarab]

Holds a shared pointer to... just the outputs?

Or the source too

[rapid hedge]

both

[slate scarab]

Oh so that basically sounds like this 😅

I wonder if those methods allocate when called otherwise, why wouldn't they just fetch them on-demand?

If they have access to the source right in the processor.

[rapid hedge]

I know why!

Because reading the reflection outputs is not safe while the reflection simulation is running

so they only fetch it between simulations

(same as we do)

[slate scarab]

who designed this API 👿

i want to talk to gaben right now

[rapid musk]

gaben rewrite steam audio in rust or there will be refunds

[slate scarab]

Anyway, I suppose we could make that wrapper, which could hold everything with an ArcGc, then then just send that to the nodes.

[rapid hedge]

aaaaaaaand it still crashes

damn

Am I holding it wrong

look, this is only place where Unity ever removes it from the simulator

Surely this is fine to do in On<Remove> then???

or does it not get disabled while a processor holds it

[slate scarab]

Hm, and does disabling it remove the audio processor?

That would represent a difference (we don't remove the whole audio processor when a source is despawned).

[rapid hedge]

OOOooooh

good idea

let me read

[slate scarab]

like i imagine SteamAudioManager is a singleton that holds all the processors (or their handles or something), and its call there will end up with it and the audio processor being dropped. Maybe.

[rapid hedge]

heh, looks like they juggle around two sources in the processor

they have one slot for the next source

then when the processor starts it thing, it moves the next source to the current source

[slate scarab]

I can't imagine we'd need to do that, though.

[rapid hedge]

Hmm I have no idea where unity sends the source over from C# to C++

it's definitely created in C#

and C++ definitely holds a shared copy

somehow

[rapid musk]

If it's created in C#, then the garbage collector will be responsible for removing it. Is there a Dispose method performing any cleanup?

[rapid hedge]

Not sure about that, I fail to find anything concrete

[rapid hedge]

Unity has its own C# dialect

I don't think they use Dispose

[rapid musk]

That's... a pretty big difference from normal C#.

[rapid hedge]

Unity has a lot of big differences from normal C#

Things that you'd expect to be interfaces are also magic methods that a unity precompiler then further processes

Using certain C# language features will break the editor

etc.

[rapid musk]

They should rename it to U#

[rapid hedge]

They're in good company, looking at Unreal's C++ dialect

and, to be fair, Bevy's Rust dialect

[rapid musk]

Bevy is still compiled by a rust compiler, which it doesn't sound like unity does.

[rapid hedge]

@slate scarab I now tried sending an audio event to the processor to plz stop processing as soon as the source is dropped, and I keep a copy of the source in the processor

still crashes.

Is the processor running while seedling drops the SamplePlayer?

[slate scarab]

Well, they're constantly running. There is no guarantee about any ordering; they're different threads

[rapid hedge]

fair enough

so uuh

can I somehow kill the processor?

[slate scarab]

Well that's not really ideal. It's slow and expensive to remove it, not to mention deferred. And we don't want to do any locking internally. Ideally we'd be able to determine the root cause so we can avoid going way out of the way to solve it

Clearly it's a bit tricky, though.

[rapid hedge]

Well I know a simple workaround that's better than segfaulting

just leak the memory for now

I assume that when I send an audio event, the very next invocation of fn process will see it, right?

I guess the processor could write some state that says "yeah I got the message, I won't work on this anymore"

and only then remove the source from the simulation

[slate scarab]

Well, this depends on the frame rate versus the audio processing rate. You can pretty much guarantee they're never equal, and one may be way faster than the other.

So several frames could pass before one audio block is processed, and vice versa.

[rapid hedge]

How do I best get a state out of the processor?

Hacky is fine

[slate scarab]

So this sounds like a garbage collection scheme to me :3

[rapid hedge]

lol yeah definitely

I'm doing very hacky things right now just to see if our ideas are correct

once I get it to not crash I'll clean it up

[slate scarab]

oh i see

i mean you can just set an atomic

[rapid hedge]

can I set one in self.params?

[slate scarab]

if the ECS sends a guaranteed unique number to the audio processor in an audio event (like with diffing), and the processor sets a shared atomic, then you can just poll the atomic in the ECS

This is precisely how I determine sample players have finished playing in bevy_seedling.

[rapid hedge]

Oh cool!

so I can just set an atomic field in the Node, got it

[slate scarab]

wellllllll not strictly no

[rapid hedge]

wait how do I read that after the sample player was already despawned

Give me something hackier

I'll just send an Arc<Mutex<Source>> into the processor

[slate scarab]

well ya you could do that

[rapid hedge]

Again, I just want to see if this stops the crash at all

[slate scarab]

the mechanism firewheel provides is state injected into the context when a node is constructed (which is supported in bevy_seedling with register_node_state)

The AudioState component will only be on the registered node though, so you'd have to follow the relationship to the node in the pool, etc. Anyway, I'll be able to investigate myself over the weekend if you don't get to the bottom of it.

[rapid hedge]

hmmmm I'm currently sending an audio event when the sample player is dropped

but it looks like the processor never receives it?

that's weird

[rapid hedge]

oh it's this then

Or nah

hm

I suppose when the sample player is dropped, any events that are sent on that frame will be dropped too

Does... does that sound right? I'm not sure

[rapid hedge]

yeah I believe I have to give up for now :/

I think you are on the right track

but I lack the knowledge to test that correctly

For now I'll just quickly merge the memory leak version onto main since that's preferable to a segfault imo

[slate scarab]

yeah if it's despawned before they're flushed (in Last), they do go bye bye

[rapid hedge]

I take everything back

yes, they do use a wrapper for the outputs as I described...

...but on the C++ side, they fetch the outputs anyways and ignore that wrapper

So uuh

that's what we do too now, I guess

managed to eliminate some 100 LOC, that's neat 🙂

and merged the memory leak so I can at least continue working on the AI integration

search for FIXME to find it

[rapid hedge]

Update: I refactored the whole AudioSource stuff and it... works now?

It doesn't crash anymore

the source is now copied to the processors when it's inserted

I'm not entirely sure what's different from my last attempt, but I'm not complaining

[rapid hedge]

is this bad?

the audio plays normally

[slate scarab]

never seen it 👀

[rapid hedge]

Happens when loading this file

[slate scarab]

maybe it was encoded weird

[cold isle]

Amazing! The plugin will make working with many objects WAY easier

Thanks, I just merged it. I'll release a patch right away

[rapid hedge]

thanks, updating the plugin now 🙂

btw, what do we do about the demo repo?

there's quite a few things we improved on the plugin repo that make parts of the demo repo PR obsolete

and at the same time, the bevy plugin has 8 examples right now

Where the glTF-loading example is pretty much just the pre-existing AN demo with a map that was loaded from a file instead of hardcoded

So I'm thinking of just closing my PR if that's alright for you

since that seems simpler than trying to keep the code up-to-date with what we are tinkering with on bevy_steam_audio

[cold isle]

I haven't had time to look into the plugin examples quite yet, but from what you're saying they seem like the right place to get started for new users as they're much cleaner, and it probably makes more sense to keep the examples in the same place as your plugin. As for the 'old' AN demo, we can probably archive it/deprecate it in favor of your repo; it'll make it clear that your plugin is the proper way to go

[rapid hedge]

thx

[cold isle]

I'm not sure if it's the recording but I think I'm perceiving a bit of a delay between the moment the sphere gets hidden by cubes and when the sound gets muffled? Or am I hallucinating perhaps? 😄 Could very well be the case, because your footstep demo sounds perfectly timed

0.9.1 is out btw 🙂

[rapid hedge]

yeah that's definitely there. I changed the occlusion algorithm and it's fine now I think. Let me record it real quick

[rapid hedge]

I still believe there is a tiiiiiny bit of delay, but that's probably due to the fixed frame size delay

[cold isle]

That's perfectissimo 👌 sweet melody to my ears

[rapid hedge]

We could in theory even remove some of that by not considering the object's visual transform, but their physics transform

since the objects are visually interpolated, they are in reality a tiiiiny bit ahead

[cold isle]

To be frank I can't notice it at all (if there's any)

[rapid hedge]

ah that's good! I know my headphones have a little bit of delay, so it might be that 🙂

[rapid hedge]

Update: forgot to plug in my headphones' cable. It was Bluetooth-induced delay

[cold isle]

Haha, you should mention that the plugin has been tested and is bluetooth-ready ✅

I just tested the example live for the first time, it's awesome! Great work! 🙂

[rapid hedge]

thanks!!!

And thank you for audionimbus and merging all my little PRs, this would not have been possible without your work

[rapid hedge]

Just updated the Bevy plugin too

[cold isle]

You're welcome, I'm stoked to see you using it! Now I'll need to work on making it safer 🙂

[rapid hedge]

We already discussed the relative costs of Steam Audio's effects, but I timed them a bit on the glTF scene to get a feeling for the actual numbers

• direct: 8 µs
• path: 30 µs
• reflections: 15 ms(!!!!)

wow, I knew reflections were expensive, but I didn't know they were that expensive

good thing they run in a separate thread 👀

Oh, also I fixed the path baking 🙂

[slate scarab]

oh are these the simulation timings?

[rapid hedge]

yep

not decoding, but running the actual sim

[slate scarab]

ya that do be spendy huh

[rapid hedge]

but you made me curious

processor timings:

• direct: 45 µs
• pathing: 22 µs
• reflect: 400 µs

let's break that reflect down...

• apply reflection effect: 300 µs
• apply ambisonic decoding: 100 µs

wonder how that will go down with the mixer @slate scarab 👀

[rapid hedge]

@cold isle I added a bunch of segfaults to https://github.com/MaxenceMaire/audionimbus/issues/29

Mentioning since last time it failed to remember any on the spot 🙂

[cold isle]

Thanks Jan, I'll make a first pass this week-end 🙂

[rapid hedge]

@cold isle I just lost an eternity to this: https://github.com/MaxenceMaire/audionimbus/pull/31

TL;DR maxOrder says "The maximum Ambisonic order of impulse responses generated by reflection simulations." but is actually also used by pathing

this resulted in access to unallocated memory in my case, but no crash -> corrupt stack with all manner of fun effects 🫠

[dusky mirage]

After copying the collector code from Firewheel for my DAW engine, I found out a way to avoid the double-indirection for boxed OwnedGc types.

/// An owned resource that automatically collects dropped resources from the
/// audio thread and drops them on the main thread.
///
/// This is similar to [`OwnedGc`], except that it avoids double-indirection for
/// boxed types.
///
/// TODO: This is a temporary workaround until
/// [`CoerceUnsized`](https://doc.rust-lang.org/std/ops/trait.CoerceUnsized.html)
/// is stabilized.
pub struct OwnedUnsizedGc<T: ?Sized + Send + 'static, C: Collector = GlobalCollector> {
    owned: OwnedGc<Pin<Box<T>>, C>,
    ptr: NonNull<T>,
}

impl<T: ?Sized + Send + 'static> OwnedUnsizedGc<T> {
    pub fn new_unsized(data: Box<T>) -> Self {
        let pinned = Box::into_pin(data);
        let ptr = NonNull::from_ref(pinned.deref());

        Self {
            owned: OwnedGc::new(pinned),
            ptr,
        }
    }
}

impl<T: ?Sized + Send + 'static> OwnedUnsizedGc<T> {
    /// Get an immutable reference to the owned value.
    pub fn get(&self) -> &T {
        // # Safety
        //
        // The underlying data is pinned into place, so this pointer is always valid.
        //
        // `OwnedUnsizedGc` doesn't implement `Clone`, and the internal `ArcGc` is
        // hidden from the user, so the only two `ArcGc`s to the underlying data that
        // can exist are the one in this struct instance and the one stored in the
        // collector. The collector never uses the data (it only drops it), and so
        // it is gauranteed that the underlying data can only be accessed by one
        // thread at a time.
        //
        // Additionally, the internal `OwnedGc` is hidden from the user and never
        // gets dereferenced, and `OwnedUnsizedGc::get_mut` borrows `self` as
        // mutable, so all mutable borrowing rules are gauranteed to be upheld.
        unsafe { self.ptr.as_ref() }
    }

    /// Get a mutable reference to the owned value.
    pub fn get_mut(&mut self) -> &mut T {
        // # Safety
        //
        // The underlying data is pinned into place, so this pointer is always valid.
        //
        // `OwnedUnsizedGc` doesn't implement `Clone`, and the internal `ArcGc` is
        // hidden from the user, so the only two `ArcGc`s to the underlying data that
        // can exist are the one in this struct instance and the one stored in the
        // collector. The collector never uses the data (it only drops it), and so
        // it is gauranteed that the underlying data can only be accessed by one
        // thread at a time.
        //
        // Additionally, the internal `OwnedGc` is hidden from the user and never
        // gets dereferenced, and `OwnedUnsizedGc::get_mut` borrows `self` as
        // mutable, so all mutable borrowing rules are gauranteed to be upheld.
        unsafe { self.ptr.as_mut() }
    }
}

Do you think I should add this to Firewheel as well?

(I was using basedrop for my DAW engine, but apparently miri found an issue with it. https://github.com/micahrj/basedrop/issues/7)

[dusky mirage]

I've decided to separate out the collector code into its own crate so that it can be used for all my projects (and for anyone wanting to use it themselves.)

[slate scarab]

Yeah I don't think there's anything too fancy that's worth doing before we get proper support for coercions.

[dusky mirage]

Published the new crate! (I've yet to integrate it into Firewheel.) https://crates.io/crates/rtgc

[dusky mirage]

@slate scarab Before I integrate rtgc into Firewheel, do you think ArcRt or ArcGc is a better name?

(Rt stands for "real-time")

[slate scarab]

I don't mind the Gc name, personally. It's immediately clear how it's realtime in that way.

But I don't have any strong opinions.

[dusky mirage]

Hmm, yeah, yesterday I was like the Rt name better, but now I'm liking the Gc name better.

[slate scarab]

it's a bit mechanistic, but... so is Arc 😅

[dusky mirage]

One last thing, I could technically avoid a breaking change if I re-export rtgc as "collector" in firewheel-core. Or do you think I should just re-export it as rtgc?

[slate scarab]

if you want to get fancy with it you could reexport it with both, and add a deprecated annotation to collector (and then remove it later)

[dusky mirage]

Actually I think I'll just go with exporting it as "collector". It's a more descriptive name of what the module does than the cryptic rtgc acronym.

Alright, rtgc is now integrated!

And do you feel https://github.com/BillyDM/Firewheel/pull/87 is ready to merge?

[slate scarab]

I'd say the only question is those cpal config types. While they may have some limited utility now with Reflect, I don't expect that to be the case after further work in bevy_seedling.

So I feel pretty good about it. That PR also added a few more derives than I recommended -- any type that implements serde's traits is a good target for Reflect.

[dusky mirage]

Ok, cool!

merged!

[cold isle]

I just merged your PR @rapid hedge (and released 0.10.0 with the fix). My oh my, I can't believe I didn't catch this before 😅 Hopefully we'll catch more of these as I write more tests and with usage. Thanks for the fix! 🙂

[rapid hedge]

Hehe I don’t blame you, it is quite niche.

That said, I also noticed your pathing tests don’t actually test pathing. You still need a bake step in there 😉

[limpid mason]

Looking forward to when Bevy has reusable graph node UI and we can build something like this with seedling: https://www.reddit.com/r/godot/comments/1okb37s/i_recreated_a_baba_is_you_song_in_my_modular/

[rapid hedge]

That UI looks so cute 😄

[rapid hedge]

@slate scarab plz check https://crates.io/me/pending-invites

[dusky mirage]

@drowsy dome Ugh, I'm having issues with GitHub Sponsors not paying out anything. I still haven't got any of the money you have donated. Could you please cancel the subscription and move to either Liberapay or BuyMeACoffee? I'm definitely canceling GitHub Sponsors once they finally pay me.

[drowsy dome]

sounds good

[drowsy dome]

which one do you prefer

I only see buymeacoffee

found u on liberapay, either's fine

fuck it venmo

naw jk just lemme know which one

[dusky mirage]

Yeah, buymeacoffe is probably more user-friendly. Though I'm not sure it's set up to accept recurring monthly donations or not.

(Ok, recurring donations are enabled).

Hmm, though buymeacoffee does take a 5% cut.

[hushed widget]

I don't know when you opened it, but they take a long time for the initial validation

[dusky mirage]

I opened it quite a while ago.

Though wait, I guess I did close it an the re-open it a few months ago.

I was thinking of setting a Patreon at some point. Though looking it up, it does seem like Ko-fi is a much better deal.

Maybe I should just take the time to set up a Ko-fi profile properly. It would be easier to just have one sponsor platform instead of multiple.

[drowsy dome]

patreon now takes 11% fwiw

[dusky mirage]

Oof

Yeah, and it looks like Ko-fi also accepts one-time donations like Buymeacoffee does.

[drowsy dome]

wow i can add a video message

[dusky mirage]

Oh yeah, I was also considering OpenCollective, since I would qualify for that for my Meadowlark project.

Maybe I should ask the Bevy team what their experience with OpenCollective is.

@drowsy dome Eh, just use Buymeacoffee for now until I figure things out.

[drowsy dome]

cool done!

[dusky mirage]

Thanks!

I think I'll also go ahead and close my Liberapay profile. I haven't gotten a single dollar from it anyway, and overall it does just look less polished than the other platforms.

Oh cool, it looks like Ko-Fi offers either a 5% cut or you can just pay $12 a month. That's nice to know since a 5% cut would be quite a bit if I end up making a livable wage off of it.

Yeah, it's looking like Ko-Fi is the best options here.

Still, I might wait until I have Meadowlark in a more presentable state before I create that. I especially at least need a website first.

[faint wigeon]

maybe @celest whale could help? i’m hoping to start one soon too so bummer to hear this

[dusky mirage]

BuyMeACoffee has been really simple to set up. To get your account approved, you just have to provide some screenshots of where you plan to add the donation links and a short explanation of why users would donate to you.

It would be kind of nice to to also be a part of Bevy's collective for my work on Firewheel, even if it's just a little bit. But of course I fully understand if funds are too tight.

I might not even technically qualify in the first place of course. Just curious how it works.

@drowsy dome Lol, thanks!

[faint wigeon]

there is no collective for bevy donations. the funds the foundation receives is governed by its bylaws. you can see the minutes for how funding is dispersed but tldr it basically entirely goes to paying alice and carts salaries as the two full time employees of the foundation.

[rapid hedge]

tried too, welp

[faint wigeon]

besides being resource constrained atm, i think there are governance problems with doing monthly support to the community. what is love to see long term is some kind of grant process for scoped work that help maintains a transparent process and time scoped funding

i definitely hope github figures things out i’d love to support you there

[hearty bear]

Github is kinda annoying since they dont accept PayPal

[faint wigeon]

i just know personally i like to minimize the number of places i'm doing donations so i don't lose track

the platformification of github is definitely bad

anyway, ot, i continue to be very excited by this work !!

[dusky mirage]

Oh actually, I remembered I created a Paypal.me link a while ago. Let me just link that. paypal.me/BillyDMdeveloper

@rapid hedge @faint wigeon ^

[rapid hedge]

that worked 😄

probably not enough to help with the computer upgrade

but know that your work is appreciated

[dusky mirage]

Right, I remember I tried to get rid of my PayPal account a while ago because they were asking me to send a photo of my ID. But eventually I just caved and gave it to them since PayPal is just too convenient to use when buying stuff online.

(Funnily they wouldn't let me close the account until I verified my identity. )

[dusky mirage]

Sweet, it worked on my end too!

[dusky mirage]

More specifically I want to upgrade my computer's RAM from 16GB to 32GB. I've been running into some issues when working on large Rust projects with multiple workspaces.

[slate scarab]

that makes sense 🤣

the crab has a thirst for memory

[dusky mirage]

I could go the cheaper route and buy two more 8GB sticks instead of two 16GB sticks, but they don't sell the exact model I have anymore and I'm a bit worried about mixing and matching a different brand.

And unfortunately the price of DDR4 ram has actually gone up since when I built my computer, despite being an older technology. 😭

At some point I also want to get a Mac Mini for testing purposes and to use as a self-hosted CI server. But that can definitely wait.

(Man I really wish Apple let to you build Mac apps on Linux)

[rapid hedge]

same, but for the exact same reason I want to upgrade from 32 to 64

I honestly couldn't imagine working with 16 due to rust-analyzed being so hungry

[dusky mirage]

Oh yeah, even more RAM would be nice.

If I find a good deal on a 64GB kit I might splurge.

[dusky mirage]

I have my parent's old laptop that I sometimes used to take notes on the road (it only has 4GB of soldered-on memory). I remember trying to play around with Bevy on it, but it just went "nope!"

It would be cool if Bevy had scripting language support.

[hearty bear]

You need a ra server

[dusky mirage]

What is ra?

[slate scarab]

i've never thought of actually running an LSP on a different machine until this very moment

[dusky mirage]

Oh, you mean rust analyzer.

Huh, I wonder if that would actually work.

Monetization idea for Bevy: SaaS subscription to use rust-analyzer and Rust compiler for low-spec machines.

[hearty bear]

Pretty sure it does. Depends a lot on how you integrate ra in your editor

But iirc vs code and jetbrains both had a plugin for it

[limpid mason]

I had that, it was not fun. As soon as you have two separate projects open at the same time, you basically go OOM

[limpid mason]

It's definitely worth it, can recommend!

Honestly, even the 32 can get high usage, but at least it doesn't crash the system anymore (on Linux) :D

[drowsy dome]

wtf how much

[dusky mirage]

How much what?

[drowsy dome]

is it ddr4 or 5?

[dusky mirage]

ddr4

[tender fiber]

You can attempt Zed on Windoze. It’s also on Mac.

[limber kernel]

have you checked out bevy_mod_scripting?

im using it currently for lua and works very nicely

[dusky mirage]

Oh I have not heard about that. I'll definitely check it out!

[dusky mirage]

Alright, GitHub got back to me and said the problem with the payout was due to me changing the email address of my Stripe account. So they just went ahead and sent me the payout!

@drowsy dome ^

Still, I think I'll still cancel my GitHub Sponsors profile. I've been trying to move away from Microsoft as much as I can anyway.

[limber kernel]

is there a preference between a "stop" notify vs a "reset" notify? "stop" meaning clear the state and pause playback, vs "reset" just clearing the state without pausing.

I see the freeverb node has the latter pattern, so was thinking maybe i keep that the same for the echo node. and then i wouldnt have to worry that notify is always triggered on the first param update, since resetting right upon creation wouldnt have an effect.

[slate scarab]

Hm, so you're still having that issue? Is the code available somewhere? I should be able to take a look in the next couple days.

[limber kernel]

ye, the visual node graph example here. https://github.com/piedoom/Firewheel/tree/echo

i took a look at the notify code and couldnt parse much but ill try and see if theres anything i missed.

[limpid mason]

Do I remember correctly that someone put together a cool demo of the spacial audio?
I wanna try it out but I forgot where to find it

[limber kernel]

i think you might be talking about @rapid hedge's stuff?

[limpid mason]

It does sound like something Jan would do

[slate scarab]

There's also the lower-level rust-audio-demo crate I've been working on, although it's very simple and could do with an update. It compares rodio, firewheel, and kira directly within a simple Bevy project.

Not very exciting, but it does serve as a point of comparison.

@rapid hedge recently drove the effort on integrating Steam Audio, which has fairly sophisticated spatialization, with Firewheel and bevy_seedling. https://github.com/janhohenheim/bevy_steam_audio

you can run the examples to see some pretty neat stuff

[limpid mason]

Ahh I believe that's what I was thinking of, thanks!

[rapid hedge]

check these!

hmm the hrtf sounds a bit flat for me

very possible I hecked up the settings, will need to double check

but the reflections sound correct 🙂

[rapid hedge]

@slate scarab is that just me or do you agree?

[slate scarab]

ya it sounds like it might not be using binaural rendering

[rapid hedge]

oh I think I know the issue

is this better?

hmm don't think so

[limpid mason]

That's super cool!
For the second one, is it intentional that the audio is entirely silenced when the emitter is occluded? In the first demo it seems to still reach the listener when you move around the corner

[rapid hedge]

yes, there's nothing to reflect sound!

[slate scarab]

yes this is binaural

[rapid hedge]

alright neat

[limpid mason]

Ohh, physics!

[wary bridge]

-# niri user spotted

[rapid hedge]

the issue is that the wall is reflecting so much sound to the listener, who is right in front of the wall, that you don't notice the direct sound

[rapid hedge]

💯

[limpid mason]

@rapid hedge do you have install instructions for the demo?
I'm getting mold: library not found: phonon when I try to run it, I suppose I'm missing some system libraries

[rapid hedge]

enable the auto_install feature

[limpid mason]

Wow ok, no idea how that's working, but I'll take it!

[rapid hedge]

hehe, it installs the phonon.so for you as part of its build.rs. Not my invention, it's a feature in audionimbus, which is the underlying steam audio binding crate 🙂

@slate scarab now this I'm happier with

[limpid mason]

I assume this won't be able to work on Wasm unless the Rust rewrite of Steam audio is available?

[slate scarab]

ya

[rapid hedge]

well, actually

there is a Wasm-compiled steam-audio

I have no clue how to link that

but I have read that others in C++ land have managed to use steam-audio on Wasm

[slate scarab]

is the rust c abi unborked on wasm yet?

[rapid hedge]

no clue

but yeah don't get your hopes up

just saying that maybe you can make it work

[limpid mason]

fair enough

Still pretty cool what y'all managed to put together in such a short time

[rapid hedge]

Thx! I'm also fairly happy with how it turned out 🙂

Just too distracted rn to write any docs lol

so uuh

let me know if you need anything explained haha

[limpid mason]

No worries, I'm not working on anything concrete for now :)
But if I was, I'd definitely try to prioritize audio, so I wanted to try it out for myself after lurking for too long :)

Very curious to see what people build once seedling is upstreamed and the steam audio stuff is battle tested

[rapid hedge]

I'm using it to power NPC's hearing senses hehe

[limpid mason]

Is it working already?

[rapid hedge]

yeah sec

[rapid hedge]

this will be first person later on, but for now top-down is easier to debug

[limpid mason]

That's cool, how well can something like this scale?

[rapid hedge]

currently about 50 NPCs

but that's for this simple scene and on my beefy machine

in a complex scene, maybe 20

but I'm planning on running the simulation code async so that NPCs can just poll their senses from there

if so, that would allow me to not hinder the main thread too much

also, I do very aggressive culling for which NPCs even get to update, so in practice you can easily have hundreds of NPCs, but that's a cheap answer since most of them are not sensing anything anyways

but yeah nothing about this setup is remotely normal 😄

At least not since Looking Glass dissolved and Eidos decided that Thief 4 would be Generic Stealth Action Adventure #5392 💀

[tender fiber]

I use Zed and disable lsp (cloud and local) in settings and it runs smoothly.

[dusky mirage]

Well, it's pretty hard to code without an LSP.

That being said, someone has generously offered me an upgraded computer. I should be getting that here before too long!

Yeah, and Zed is a cool text editor. I just happen to like the workflow of VSCodium a little bit better (especially when it comes to search and replace). Also VSCodium has a lot more themes to choose from. (The Zed ecosystem is severely lacking in themes with neutral gray background colors. There are a few, but they aren't that good.)

Though I do respect that turning off the AI features in Zed is as simple as clicking a single toggle button in the settings.

[tender fiber]

I’m glad.

[vagrant bay]

Hi everyone! I'm Juan, sound designer for some years coder for fewer. Just wanted to join the team to see how we can make Bevy audio system better with some features that could help devs explore new possibilities with audio. We can use WWise and FMOD for inspiration although I think that our design should be aimed to devs with no audio training.

[celest whale]

Take a look at https://github.com/BillyDM/Firewheel and https://github.com/corvusprudens/bevy_seedling

This group is currently refining those crates, and I would really value your feedback about their design and usability @vagrant bay

[vagrant bay]

Oh, this is good! Didn't see it on the website or the source code I checked so I thought it didn't exist

[celest whale]

Yep, it's not currently adopted 🙂

[vagrant bay]

Thanks for letting me know about them!

[celest whale]

I'd like to see a head-to-head comparison against Rodio still, but I'm not sure what @slate scarab et al have planned there 🙂

[vagrant bay]

How can I help the audio team @celest whale?

[celest whale]

User feedback on those crates, then a demo to show the ecosystem and maintainers, then once we're sold that it's better, a PR to swap out the current bevy_audio crate for the contents of bevy_seedling

[vagrant bay]

Awesome! Thanks for giving me a bit of direction

[celest whale]

That's my job 😉

[slate scarab]

Yeah probably one of the most helpful things to do at this stage would be to evaluate the APIs and overall capabilities. The sooner we land an integration of Firewheel, the more opportunity we'll have to work towards FMOD-like capabilities!

Audio work tends to be more "artisty" than "engineery," if you don't mind my broad strokes, so I think we'll have fewer eyes on the audio side of things for a while.

[celest whale]

There are also a lot of very hard constraints and relatively niche knowledge involved

In a way that's not really true for most other domains (rendering excepted)

[vagrant bay]

I understand, hopefully I could take care of the audio aspects of the engine, devs could enjoy more working on this field instead of worrying about not knowing how to implement it.

Anyways, I'd like to start taking part of this on Monday by learning Bevy basics first and make a very small video game to use it as a sandbox.

[dusky mirage]

Yeah, definitely. I would not have been able to create Firewheel if I haven't already spent multiple years passionately working on other audio programming projects. (And even then I still learned a lot of new things in the process of creating Firewheel).

I guess I am a bit more interested in the "engineery" side of things. (As in I like making the tools that others can use to do more "artisty" things.)

[vagrant bay]

Alright everyone, just wanted to let you know that I had a look at bevy_seedling and firewheel repos. They cover a considerable amount of aspect for implementing audio however I can't give my opinion until I code a bit to evaluate two things: 1. My experience using the apia and 2. The end result.

We want a demo and I want to make one. I started to get familiar with Bevy and its ECS paradigm which is fantastic for those like me who love to work with terminals and hate bloated software. The fact that you can choose which modules to use is a dream true to me.

To become an expert on it I'd take lots and lots of time but we need to move forward. For that reason I'm gonna follow this pong tutorial. My idea is to implement audio bit by bit so show the team project and open the results to discussion. Please let me know your thoughts about this plan

[slate scarab]

On the demo in particular, I believe the intent is to evaluate the baseline capabilities of the relevant engines. These would be rodio, kira, and Firewheel.

I have such a demo for the engines here. It was based on the specs laid out here in the Rust Audio Ecosystem group.

I think there was some interest in making it a bit more of a stress test, but it's actually a lot of work to put something together that puts strain on any of these engines and makes sense in the context of a demo. Consequently, neither I nor anyone else has taken it farther.

I have done some very simple stress tests on my own involving the three engines, and Firewheel seems to consistently comes out on top! But I haven't done anything formal.

[slate scarab]

In any case, I would still very much encourage this exploration! We very likely could use higher-level comparisons between the engines using their respective integrations (i.e. bevy_audio vs bevy_seedling vs bevy_kira_audio).

[slate scarab]

I think an ideal shootout would involve extensive, carefully crafted, ECS-first integrations for each engine. This would more clearly illustrate how well each meshes with the ECS. But, if I may, only one has received such an integration ;)

I think you'd fundamentally struggle to get much expressiveness out of rodio and the ECS even as it improves. kira could be reasonably nice I think, though it might require more newtypes than we'd prefer.

[rapid musk]

Is the demo the only thing remaining before considering swapping out bevy_audio with bevy_seedling? Benchmarks and user feedback is giving me the impression that bevy_seedling and firewheel is the obvious choice at this point. It makes me wonder what there really is to learn from having a demo comparing seedling against rodio or anything else. It's a cool idea and the demo might excite users who haven't been paying attention to this work group, but that doesn't seem like a reason to block further progress

I'm assuming the demo isn't something that will be maintained, so it's usefulness in teaching how seedling works will also degrade over time

[slate scarab]

Is the demo the only thing remaining before considering swapping out bevy_audio with bevy_seedling

Not necessarily I think -- it's just something that will help motivate upstreaming.

[rapid musk]

Hmm, are there any features missing from seedling/firewheel that are in bevy_audio? Or are there any other major remaining work items in either seedling or firewheel?

[slate scarab]

I would say the answer to both of those is a firm "no."

I think both crates could be refined, and there are a number of audio processing tools we can add to Firewheel (including more sophisticated reverb, filtering, and dynamics processing). But the same can be said for kira and rodio, and you can't really work with effects at all in bevy_audio right now.

Probably the highest-impact item remaining is audio streaming, but this intersects a bit with asset streaming in general (which we don't have), so I don't think it represents a real blocker.

[rapid musk]

I agree.

If we're just waiting for more user feedback, then I worry that it will take a long time for users to find out about bevy_seedling organically. Would it be possible to get an announcement through official channels that bevy_seedling is being considered for upstreaming and user feedback would be greatly appreciated? Perhaps a post in #announcements and/or news post?

[celest whale]

The user feedback has been positive enough that I just want a doc with a feature comparison matrix and rationale for migrating + a modest demo

Something that the maintainers can evaluate, and something we can use as a base for the release notes

And ideally something that firewheel can use to create marketing material out of

[rapid musk]

Perhaps the demo that @slate scarab has made is already good enough then. Do you think the demo should also be stress testing the different solutions? It was mentioned earlier that this is a lot of work. Or is it purely for evaluating the different APIs against each other, and perhaps show off a thing or two that seedling can do, that the others can't?

[slate scarab]

I think it's both too simple (no graphics isn't the best marketing!) and it also simply compares the underlying engines. If we want a tour of the upstreaming candidate, then it doesn't make sense to leave out bevy_seedling.

I think the motivations for the demo that I was writing are a bit different from what would be useful for Bevy. The guidelines I mentioned came from the Rust Audio Ecosystem effort, but activity seems to have stalled there as it often does.

As I mentioned earlier, I don't personally think a stress test is all that valuable. Firewheel has been consistently ahead in my testing and scales very well, but the delta between these engines is on the order of 1.5-3x under normal conditions. I think most devs won't run into any performance issues regardless of the engine (excluding main-thread Wasm problems, which will plague rodio until cpal merges web multi-threading).

With that said, it sounds like the next steps for this working group are

1. Consolidate the original design doc and restate the rationale for this effort.
2. Write up a feature matrix comparing bevy_audio and bevy_seedling. Alternatively, we might compare rodio and Firewheel, imagining what an optimal integration could look like for each.
3. Create a modest demo specifically showcasing bevy_seedling. This would allow us to demonstrate the massive leap in capabilities between bevy_audio and bevy_seedling, rather than sticking to the set of features they both share.

(1) and (2) are pretty easy, but (3) represents a notable investment. Presumably, we'd like a demo in 3D for the best demonstration of the spatialization capabilities. Furthermore, I imagine we'd want a reasonably high-quality soundscape and some occasional interactivity.

I've personally worked almost exclusively in 2D, so that's a bit of a barrier for me. The extent of my interactions with 3D gamedev actually come from collaborating with @rapid hedge, funnily enough!

If you have any thoughts on the matter Jan, I'm all ears! Maybe creating a 3D environment (particularly with Bevy) is easier than I think

[cerulean shoal]

A feature matrix would be really nice to see because as someone that doesn't really do audio it's not entirely clear to me what firewheel brings to the table other than performance

[celest whale]

And a lot of non-audio devs don't understand exactly what "performance" means in the context of audio

[rapid hedge]

Oh I can easily whip up a 3D demo if that helps 😄

Just give me the instructions of what you want

e.g. a big museum-like collection of rooms where each room shows a node

(Assuming such a demo is allowed to depend on Avian and TrenchBroom)

[slate scarab]

ohh that would be so fun!
i should have time soon to start working on the rationale and feature breakdown, which will help orient a demo

[slate scarab]

Yeah I assume this wouldn't serve as a minimal tutorial or example, in which case additional dependencies seem perfectly reasonable (and a real time saver ;).

[rapid hedge]

I can imagine e.g. a room with a spatialized source orbiting around so you can hear the HRTF, another room doing a lot of reverb, etc, controlled by invisible triggers

And maybe some podiums with buttons if needed

Explanatory text can just be on the HUD when you enter the room, that seems easiest

Maybe the room for the LPF and co could allow you to tweak the frequencies with some simple buttons

[rancid elm]

maybe you could use your hl2 pickup thingy to control audio based on the position of some boxes or something

[rapid hedge]

hehe that's a neat idea

[rapid musk]

Implement doppler effect and shoot ambulances past the players right and left ear 😛

[thick orbit]

or more practical - bullets/debris flying by

[hasty lichen]

if I wanted to get a jump start on using spatial audio in 3d in a somewhat future-compatible way.. should I just switch and use bevy_seedling now (does it have spatial stuff)? or the steam thing Jan worked on (is that separate)?

[rapid hedge]

yes, use bevy_seedling

bevy_steam_audio is implemented with bevy_seedling

so it's not either-or 🙂

from a user perspective bevy_steam_audio just gives you some more fancy audio nodes to play with

the biggest difference is that bevy_seedling's builtin stuff has no concept of walls, while bevy_steam_audio does raytraced room simulation for reflections and stuff

[hasty lichen]

that sounds expensive haha. I'm checking out bevy_seedling now and it seems pretty great

[rapid hedge]

It is! That's why it's split into 3 parts:

• cheap direct audio
• medium pathfinding
• expensive raytracing

direct audio is cheap enough to run every frame and is already enough to have walls block audio

then in a separate thread, you run the other two and pull in their results whenever they're ready

bevy_steam_audio does this for you under the hood

[hasty lichen]

I'm really just looking for cheapest spatial possible.. Like, I'm ok with just volume being adjusted by distance. is that just the basic bevy_seedling spatial audio?

[rapid hedge]

yep

if you don't need walls to block audio, bevy_seedling is more than enough

bevy_audio could in fact also do that, but... don't use it

[hasty lichen]

(the videos you've posted with wall blocking/reflection/reverb sound incredible)

[rapid hedge]

really, it's a headache

[dusky mirage]

You can also just set the "damping" parameter in the SpatialBasicNode to a higher value to make something sound like it's coming from behind a wall. And then use the ConvolutionNode to add reverb. It's not as fancy as the steam audio stuff of course, but it gets the job done.

[slate scarab]

Yeah in other words, you could pretty easily perform a basic occlusion check from source to listener. Like with a raycast.

[hasty lichen]

does the SpatialBasicNode have to be on the root of an entity? I was setting up something where I had multiple samples as child entities to my player entity but when I do that, I don't get any spatial effect, it is just always the same volume. If I spawn the same effects without parents, I can hear the spatial effect (at the world origin)

[slate scarab]

Do those children also have transforms?

[hasty lichen]

that was it. That makes sense, I think I got it confused with the spatial listener which has a require transform attribute, but the spatial node~~ is an effect and not a component so that makes sense. Cool 👍~~ oh wait, it is a component it just doesn't have the required transform attribute. Maybe it should? 🤔

[tender fiber]

It has x and z transform no y in its calculation so sprite x,y transform is not satisfied. I wanted a vector_2d to impl x,y and was shut down (explained horribly by me)

Also, I wanted to not bother people at this time of year.

[dusky mirage]

Oh yeah, I suppose it would be pretty helpful to have an option to choose the axes of the SpatialBasicNode. That should be really simple to add. (Once my WiFi comes back online that is.)

Oh, maybe I could try using my phone's hotspot feature.

Hmm, it would technically be a breaking change though.

Oh actually, flipping the y and z axes won't make any difference. They both only contribute to the distance calculation, so they are equivalent.

I should make that clearer in the docs.

[dusky mirage]

Ok, I made the docs for the SpatialBasicNode clearer! https://github.com/BillyDM/Firewheel/commit/1aa78242f1d2dba0040b3ba98771f49a1864eb0e

[frank bluff]

So, migrating over here from #general ... I'm pondering using bevy to make the user interface for a modular synth type thing. The synth engine is largely working - it (plus the interface it uses for audio and midi, built around jack) happily run in their own set of threads, with communication via channels/ringbuffers as appropriate (so, thread safe and self-contained). I'm thinking about bevy to build some kind of interface to it, as it needs to be quick and interactive, and visually it seems like it might be interesting to use 2d/3d rendering to build the interface rather than just a ui/widget approach.

As much as anything I'm wondering how to use things like channels/ringbuffers in a bevy sense - I did find one example using channels, but it only read from the channel at fixed intervals, which would definitely not work - it would need to be "reactive" in a sense to changes from the engine (so I guess it would need to read state changes on every frame update).

Just wondering if other people have tried something similar, or have experience thinking in that way! It's definitely an audio/daw type engine with a game-type ui idea, rather than a game audio engine, so Firewheel probably isn't directly relevant - except that obviously a lot of thought has gone into it and there's probably been a lot of learning about what works!

[oak walrus]

Firewheel was a DAW audio engine first before becoming the backend for bevy_seedling and the eventual upstreaming into Bevy, so it's certainly somewhat fitting!

The problem I see is if you want to make it into a VST3 or CLAP plugin, then Bevy is going to struggle as it can't be used by default for the GUI in that context

[frank bluff]

Ah, I'm not even slightly bothered by turning it into a VST3 or CLAP plugin! 😂 I know that might sound odd, it's just not a goal... It just feels like an ECS might be a good fit for a dynamic environment, and if I can work out a good way to integrate engine and bevy, it opens up a lot of potentially nice things. I guess my challenge is very much of the "how do I set the engine up in the background, and then communicate potentially quite a lot of data from it without slowing down either the engine or the game loop"!

I guess being more concrete, in one direction I'm going to have instructions from the interface going to the engine (create a new module, set this property to this value, and so on). And on another, potentially data going between modules and the UI, maybe just as simple as "this light should be on" but also potentially something like "show this live waveform" which would be a bit more data - but maybe not too much, if I can make sure that channel/send receive works without ever blocking/allocating on either side...

I guess this is actually a pretty reasonable example of what I'm trying to do in an abstract sense: https://github.com/bevyengine/bevy/blob/main/examples/async_tasks/async_channel_pattern.rs although whether I'd want a single channel pair or typed channels per module type/instance is a different question, but I guess it's possible to do that kind of thing in bevy if I needed to!

[frank bluff]

I'm basically trying to make the engine as self-contained as possible at the moment, and with any luck that'll make integrating with bevy a bit easier...

[deft jasper]

out of curiosity, what's the issue with being a vst plugin? winit?

[dusky mirage]

The processor part of the Firewheel engine (the part that lives on the audio thread) is single-threaded only. Doing multi-threaded processing on the audio thread in a way that is low-latency enough for realtime audio is a very complex topic. (Most modular synthesizers don't even bother with multithreading).

[dusky mirage]

Well, sort of. The goal of Firewheel is not to become a fully-fledged DAW engine to keep the scope manageable. Firewheel only handles audio buffers and GUI->processor events. If you want more features, you'll have to implement them yourself.

DAWs can get away with multithreaded processing because plugins generally take a lot more CPU than basic nodes in a modular synth, so the overhead of thread synchronization is less of an issue. (That, and the fact that plugins in a DAW tend to be quite sequential per track, making it easier to assign a single worker thread to an entire track.)

[dusky mirage]

I just had a thought, should Firewheel replace log with tracing? That would be a breaking change though, so it would need to be published in a new Firewheel version.

[celest whale]

Take a look at Bevy's strategy. IIRC we use a fallback mechanism with feature flags

[deft jasper]

There's some desire to drop log from bevy as well, but it looks like tracing isn't going to support nostd until 0.2 releases: https://github.com/bevyengine/bevy/pull/20539

https://github.com/tokio-rs/tracing/issues/3413#issuecomment-3666705675 is the tracing comment referencing that

[oak walrus]

Yes. In a plugin environment, the DAW owns the event loop, and the plugin hooks into it. winit does not like this, and it's a non-goal of the project to make it work in hosted environments. You need another windowing library that can be hosted. baseview is one such project that's used in Rust audio plugins. If you can cleanly replace winit with baseview as the windowing layer, you can use Bevy in a plugin

[deft jasper]

that makes sense, thanks!

[oak walrus]

Actually, with Bevy windowing being abstracted away from winit, it might actually not be too much work to have Bevy run a plugin GUI

[frank bluff]

Yeah, multi-threading can be a pain, but I'm reasonably comfortable with my current implementation which effectively shards processing - generally speaking it would operate single-threaded as that's most efficient up to a certain utilisation level, but you can add more shards(=threads) if you're working with patches of significant size, and in that mode it'll distribute modules to the least utilisied shard, etc. It's all experimental, but we'll see how it goes!

The engine is intended to be entirely standalone - it should be possible to build multiple UIs for it, from something like a live code lang, to a tui app, to something more fancy - hence thinking of bevy for something very visual. If I can find a nice comfortable model for interaction and state visualisation, I think it could be a reasonable fit.

At the moment my struggle is still on the level of "what should be a resource, what should be a component, what should live in bevy-ecs land and what should be a plain old rust lib" etc., as I don't yet have an intuition for things in bevy-land (and I've never worked in gaming, so I'm starting from scratch with a pretty unfamiliar model!)

[dusky mirage]

@slate scarab Would there be any issues on your end if I updated Firewheel to cpal version 0.17.0?

[slate scarab]

Hm, well we could just use the opportunity to update for Bevy's 0.18 rc at the same time.

I don't think I depend on cpal in any way that would be broken, though.

[dusky mirage]

Ok. I'm currently in the process of adding tracing support (with log as a fallback) which is a breaking change anyway.

[dusky mirage]

@slate scarab Oh yeah, we should also think about upstreaming your CPAL webassembly stuff into the Firewheel repo.

[slate scarab]

they actually already merged their own solution

[dusky mirage]

Oh, ok. Cool.

[slate scarab]

so it should "just work" with the required feature soon

looks like you need wasm-bindgen and audioworklet

[dusky mirage]

Sorry I've been away for a bit (the holidays have been busy, and I also haven't been feeling that well.) But now I'll try to get Firewheel updated with the new cpal version and with the new tracing stuff.

[dusky mirage]

@slate scarab I have a branch up for the updated cpal API and tracing support. Let me know if you run into any isues with it! https://github.com/BillyDM/Firewheel/pull/96

One last thing I want to do is move the symphonium stuff out of firewheel-core, so that updating symphonium won't be a breaking change for nodes that don't use it.

[dusky mirage]

@slate scarab Ok, I also moved the symphonium stuff into its own crate module. (This is a breaking change with firewheel-core's feature set.)

[slate scarab]

I think the Bevy 0.18 release is pretty close. If you want, you can probably wait a little bit to tick those Bevy versions and publish a new set.

(the rc is currently in progress)

[dusky mirage]

Yeah, that's a good idea.

[slate scarab]

I think it's only a few days away if all goes well

[dusky mirage]

I'll just go ahead and merge the PR into main. We can figure out any issues when updating it to the latest bevy version in a separate PR.

[dusky mirage]

I've been trying to integrate the alternative RtAudio backend into Firewheel. Turns out the API is quite different than CPAL, so I think I need to find a better approach than the AudioBackend trait.

[slate scarab]

Where do you feel the pain points are off the top of your head?

[dusky mirage]

Essentially, they way I designed it, the backend has to create a new "host" object for each method. This isn't a problem with cpal, but I'm having problems with rtaudio. I don't want to have to resort to using a static singleton object.

[slate scarab]

Coincidentally, that's what I did for the web audio backend since its initialization is async.

Actually, no that's not true. The success or failure just can't be communicated directly in the start_stream method.

[dusky mirage]

I think the cleaner solution is to just have the backend define a custom "device enumeration" type.

[slate scarab]

Do you think it's possible to capture all requirements in a single, concrete type? That would be great for the ECS API anyway since all backends could use the same interface.

For devices at least.

[dusky mirage]

Hmm, well CPAL and RtAudio expose different device parameters (for example RtAudio has an addition "duplex channels" parameter).

[slate scarab]

I suppose I could always set up type erasure at the ECS API level anyway.

[dusky mirage]

And I'm not sure how useful RtAudio will be for games anyway. The only reason to use it is if you need native duplex support to have lower latency between the audio input and audio output.

And it also limits you to only desktop platforms.

[slate scarab]

I could see it being useful for voice processing, like applying reverb to a player's voice in a multiplayer game. I bet this is rarely done in part because the latency is typically difficult to manage.

I think a number of non-game Bevy people are also very interested in low-latency processing.

It's definitely tricky though. Abstracting over pretty different approaches like this seems to get messy no matter how hard you try.

[dusky mirage]

Ah, ok. Well for games at least, it's not common to play back the input directly to the output, because hearing your own voice with any kind of latency (including the latency inherent to the OS's audio thread), is very jarring. The preferred way to play back your own voice is to do it in hardware to achieve zero latency (which is a feature most modern audio interfaces have).

Yeah, I think having the backend define a "device enumerator" struct is probably the cleanest solution.

[dusky mirage]

And if you're doing reverb/echo, there will be a delay anyway.

[slate scarab]

Hey you could probably pull it off if you call the latency "pre-delay"

[tender fiber]

He used cpal DeviceTrait, HostTrait, and StreamTrait to API into cpal. I think the harder part is the API to the backend. I was only able to fork cpal to API into backend. Also, cpal is very picky so making sure cpal likes new API should be starting point instead of great API cpal hates.

[dusky mirage]

@slate scarab Alright, I've finished adding an RtAudio backend to Firewheel and reworking the AudioBackend API! https://github.com/BillyDM/Firewheel/pull/97

One more thought I had is that it it's not the most ideal that firewheel-core depends on bevy_platform, since it will be a breaking change every time bevy updates. Got any ideas how we might get around that?

[slate scarab]

Hm, do we expose anything from bevy_platform that would force a breaking change? The time stuff perhaps?

But either way, I don't think it necessarily needs to be updated with everything else. Unlike bevy_ecs and the Component implementations which simply won't work, bevy_platform at worst would end up with two compilations if it's not updated. Not necessarily ideal, but not that bad.

[wary bridge]

I bet we can probably do one of two things:

• If bevy_platform has been pretty stable, go ahead and push it up to 1.0
• If not all of it is stable (I imagine most of it is), pull the stable parts out into a separate crate, reexport it from bevy_platform, and push that to 1.0

[celest whale]

We can stabilize bevy_platform ahead of the rest of the engine

[dusky mirage]

That would be awesome!

[celest whale]

cc @stable berry, since you've been considering partial stabilization

[celest whale]

yeah a layer of documentation and surface level polish on the crate would do wonders, it looks very high effort

Feedback about firewheel from a dev I introduced it to, for use outside of Bevy as a rodio alternative

[dusky mirage]

Oh yeah, for a while me and the rodio devs were talking about basing rodio on Firewheel, although I don't think anything came of that.

And yeah, I think Firewheel is almost in a stable enough state to start polishing it (including maybe a website). (I'm just waiting for bevy_platform to stabilize and @slate scarab to finish the Bevy 0.18 integration). Though I'm not sure how much time I should spend on the polishing vs working on my other projects. If anyone wants to help with that task, I would really appreciate it!

At some point I would also like to get the C bindings integrated, since that would potentially bring a lot more interested users.

[dusky mirage]

(It also ended up being much higher effort than I originally anticipated when I started Firewheel. 😅 )

[celest whale]

It always is ❤️

[dusky mirage]

I'm also debating on whether I should move Firewheel to Codeberg like my other projects, or to keep it on GitHub.

I'll need to figure out the CI problem before I do that. Codeberg has a "2 minute limit" for their free CI, which is a bit discriminatory towards Rust projects. 🥲

And plus Codeberg doesn't offer Windows or MacOS CI.

I've also been considering just buying a Mac Mini and turning it into a self-hosted cross platform CI server.

(As well as allowing me to test software on Apple Silicon and MacOS)

[stable berry]

You are of course free to do whatever you want, but from a collaboration / Bevy-community-engagement perspective, being on GitHub would be functionally better

(I do love Codeberg and what it stands for)

Ex: we can reference issues from each other, mention collaborators, etc

[dusky mirage]

@slate scarab In regards to the PR https://github.com/BillyDM/Firewheel/pull/97/, were you thinking of having a generic method that just returns a list of device names to use?

Both CPAL and RtAudio have different implementations for DeviceID, so that unfortunately has to be generic. And as for getting device capabilities, CPAL technically does the right thing by separating getting the device ID and getting the device capabilities, where as RtAudio just scans the IDs and capabilities of all devices at once which is slow on some backends.

[slate scarab]

I'm not sure, to be honest. It's quite annoying! Platform abstraction is one of the things I really hate about programming.

I think what would be ideal for bevy_seedling is a fully type-erased device where names and capabilities are all methods that return options. I don't think that's ideal for Firewheel, though.

I'll probably need to do this in bevy_seedling at the end of the day. Right now my backend handling is only partially abstracted -- you need the concrete stream configuration type to adjust I/O anyway. So I'll probably need a sweeping abstraction for the backend and all its devices.

As I mentioned, this is annoying for people who want to pull in third-party Firewheel backends due to the orphan rule. But in practice, I think this will never really happen. I'd have feature flags in bevy_seedling for the relevant backends and implement this abstraction for them directly. And anyone with a custom backend could implement the abstraction themselves.

To be clear, by "abstraction," I really mean a trait with methods like set_input, set_output, get_inputs and get_outputs that operate with type-erased data.

[dusky mirage]

Yeah, it's a tough question of "how high/low level should Firewheel be?". I suppose I consider Firewheel to be like "wgpu but for audio", so I tend towards the low level end.

I see.

[slate scarab]

Then again, I may not need that at all if I lean into the ECS.

If I write backend integrations as Bevy plugins, I could likely skip the middleman abstraction trait.

[dusky mirage]

Another problem like we discussed before is that CPAL and RtAudio return different information about the devices.

[slate scarab]

Yeah, that's where the ECS could really come in handy. Device information could be components, and then the abstraction becomes a lot more elegant.

[dusky mirage]

The information does have a lot in common though, but again I want the user to have the ability to get access to all of it.

[oak walrus]

I've been thinking of the platform abstraction for a while with interflow, and I kinda came up with the answer that abstracting away configuring devices is kind of a catch 22. There's enough overlap that intuition keeps telling me there's a way to make it work, but just not enough that every time I think I have a solution it's not great.. I think there's a reason why all the libraries of this kind are "lowest common denominator". For interflow I think what's gonna happen is that only the audio stream callback is going to be abstracted, with an additional type-erased, no configuration abstraction for the case of quickly getting going, which might satisfy most people but then if you want platform-specific configuration, you can use the platform-specific types

[slate scarab]

Yeah I think an escape hatch will be mandatory. In other words, you should be able to downcast or get access to the raw types if you need them.

[oak walrus]

For an ECS it might basically be the same thing. As long as you end up with an entity that has the component to start an audio stream, you don't need more as an integration point for seedling, but people might want to lean on specific platform features

[slate scarab]

I'm curious if anyone knows whether there's prior art with other platform abstractions in Bevy. Maybe not, especially since "everything as entities" seems like a more recent design trend.

[celest whale]

Winit and wgpu are the main places to look 🙂

[slate scarab]

We leave the abstraction to those crates though, don't we?

[oak walrus]

I was gonna approach it from the other way around, as in you only use concrete type, especially since you can have different platforms within a single OS you need to be able to make that choice

[slate scarab]

Hm, I see. Although I'm beginning to think that abstractions written in terms of entites and components may overcome this.

[oak walrus]

winit is not a good example because there is no choosing a backend, it is given by which OS or windowing system you're using

[celest whale]

Mostly. You can also access the wrapped platform specific APIs directly via a bit of an invocation, to allow power users to do weird platform specific things

[oak walrus]

wgpu is more interesting though because it has capability negociation and extension points

[celest whale]

Yes, I really like the capabilities model here, especially when exposed at runtime

[oak walrus]

Hm, what if interflow had an extension system? Like clack for CLAP plugins, you could create extensions and query then on the backend

[slate scarab]

The only awkwardness with interfaces written in entities, components, and events is that you can't necessarily write normal procedural code. To fetch devices, you'd trigger an event and then wait for them to show up (as in resume behavior in another function).

[oak walrus]

sounds like async event handling might be a nice to have

[dusky mirage]

Though I do think it would be nice to have an easy-to-use interface to handle the most common use case of just getting a list of input/output device names and then selecting them.

[slate scarab]

To expand on my thinking here, the ECS representation could look like this

let backend = commands.spawn((
    BackendMarker,
    SampleRate(48_000.0),
    // we could put additional information here like
    // platform APIs (Alsa, Jack, etc.)
)).id();

let output_device = commands.spawn((
    OutputDeviceOf(backend),
    // Let's say this is the default
    DefaultDevice,
    Name::new("Speakers"),
    // We could insert backend-specific IDs
    DeviceId::new(rtaudio_id),
    // And any other platform-specific data
    DuplexChannels(2),
)).id();

// selecting a device could be as simple as
// establishing a relationship
commands
    .entity(output_device)
    .insert(SelectedOutputDeviceOf(backend));

// alternatively, selection could be triggered by an event
commands.trigger(SelectDevice(output_device));

Is this too far? Or does it elegantly manage the differences you'll see between backends? It certainly allows you to not care at all about the backend if you don't want to, which I personally think is a great advantage.

@faint wigeon I know you're a fan of leveraging the ECS. Do you foresee any papercuts with an approach like this? (Let me know if my suggestions here aren't clear.)

This is more or less the logical conclusion to the approach I've begun to take in bevy_seedling. It's currently half-baked.

I suppose I'd fill out the suite of abstractions with events like StartStream, ResetStream, and so on.

[faint wigeon]

this looks great. my only question would be mostly my own ignorance about how cpal/firewheel works. we configure sample rate on Backend here. this is basically equivalent to wgpu where Backend is a "logical" device and Device is a "physical" device? so if i had two audio interfaces i'd have 2 backends?

[slate scarab]

I think the backend would represent the Firewheel context (which is generic over its backend). The context manages the audio graph and does the processing. The context can have one "stream," with an optional input device and output device.

Technically, there's nothing stopping you from having multiple Firewheel contexts in a single application, but I think we'd make it a singleton for now, and ideally a resource once resources are entities. (It would become ambiguous which context you're targeting when spawning bevy_seedlings's sample players and processing nodes.)

The advantage of the entity representation would be the relationships.

[dusky mirage]

Here's one idea for an "easy mode" interface.

The AudioBackend trait in Firewheel could define get_input_devices(&mut self) -> Vec<BasicDeviceInfo> and get_output_devices(&mut self) -> Vec<BasicDeviceInfo> methods:

pub struct BasicDeviceInfo {
    pub name: String,
    /// The device ID serialized into a string. (Both CPAL and RtAudio support serializing/deserializing from a string without the use of serde).
    pub id: String,
}

And the first device in the list will be the "default" device.

And then you have a generic stream config defined as:

pub enum SelectDevice {
    /// Use the default device,
    Auto,
    FromID(String),
}

pub struct SimpleDeviceConfig {
    pub device: SelectDevice,
    /// Set the number of channels to use. The backend may end up using a different
    /// channel count if the given channel count is not supported.
    pub channels: usize,
}

pub struct SimpleStreamConfig {
    pub output: SimpleDeviceConfig,
    pub input: Option<SimpleDeviceConfig>,

    /// The block size (latency) to use. Set to `None` to use the device's default
    /// block size. The backend may end up using a different block size if the given
    /// block size is not supported.
    pub block_size: Option<usize>,
}

And then the AudioBackend trait will have a convert_simple_config(&mut self, config: SimpleStreamConfig) -> Self::Config method.

I think that would handle the majority of use cases that games would need.

Hmm, though I suppose it would also be handy to have the number of channels in BasicDeviceInfo, so that the user can select a surround sound device for instance.

And maybe SimpleDeviceConfig::channels could be an Option<usize>.

[slate scarab]

But then we're back to eagerly fetching device info aren't we?

(Which may very well be fine for a simplified interface.)

[dusky mirage]

Ah yeah.

Though thinking about it, the number of channels could be something that the game gets after the stream is started.

[faint wigeon]

here's a real world use case that might help orient this discussion. i have a demo installation where i have the following audio i/o:

• a system mic input on my laptop
• an audio interface outputting sound events on a stereo 48000 channel
• an separate AVB interface outputting i16 samples at 96kHz for a laser

we want to have the mic input do some basic processing to output events when it hears something like a clap in the room
these events trigger some audio sample
separately we do some animation on the gpu that is readback to feed the laser output

my understanding is that in this setup, we don't need any kind of audio graph between these three separate devices which can all be routed through the ecs, but i'd like to understand if it would be possible to configure / manage these devices

[dusky mirage]

Yeah, StreamInfo already contains information on the number of input and output channels.

So do you think I should try adding this, or should we continue looking for other options?

[faint wigeon]

also again i don't want to make y'alls life hard obvi lol but i'd love if it were possible to support these use cases ❤️

[slate scarab]

I’m not sure yet personally.

[slate scarab]

Hm, I’ll have to think about this for a bit.

I think this could be done, but it seems like it would demand either multiple Firewheel contexts in a single world or multiple worlds each with one context.

[dusky mirage]

You can also just spawn another CPAL stream in addition to the Firewheel one.

(Or RtAudio Stream)

[slate scarab]

Right, we could have device management that’s independent of a Firewheel context.

Although that doesn’t seem very useful (you’d have to use the backend APIs directly to do anything with them if you don’t use a Firewheel context).

I like this as a concrete use case though. Multiple contexts in one world is very achievable.

[faint wigeon]

just as a more general comment, i've noticed this across a variety of different contexts with bevy plugins when interacting with hardware (including bevy render gpus). i think there's a temptation to start with the api that you configure a selected hardware device when adding the plugin to the application. this is ergonomic because for most applications it's a piece of static config that's part of defining the application. it'll never change and doesn't need to be introspective

but inevitably, there's a lot of more dynamic things that can come up: what happens when that hardware is unplugged? what if i want to make a menu to select it? what if i want to have 2 of this thing?

i'm going through this right now refactoring https://github.com/mosure/bevy_webcam to be able to support multiple webcams.

and i get the tension, because the plugin-as-config model also allows you to use resources as singletons, which typically leads to the better api for the 99% case where you just have a single instance of the hardware thingy.

but i do think the fully componentized design is probably also better for evolution and supporting future edge cases as they come up

but like we are very much going to go through this evolution with gpus, where we move it as much as possible into the ecs as components that can disappear and hopefully recover when the device is lost, etc

[slate scarab]

I'd say bevy_seedling is well positioned to move the context into an entity. The main question would be "how do you associate an audio node with a particular context." Of course, I've already answered this question with SamplePlayer and the desired pool (without a label, it gets queued in the default pool if it exists).

In other words, I'm fairly certain this move could be made without much breakage and without making the easy path any more complicated.

[faint wigeon]

totally, and i mostly want to bring up these use cases just to make sure we don't create any apis which would make it significantly difficult to make changes to enable this kind of thing in the future. i'm still very in favor of getting seedling upstreamed in whatever form and iterating from there

[slate scarab]

Okay @dusky mirage, sorry for slowing you down!

Long-term, I'd like to build up a more cohesive abstraction controlled directly by bevy_seedling so I can design it in the vocabulary of the ECS. However, I don't think it's worth pursuing right now.

That means I would really appreciate your suggestion earlier about a simplified I/O scheme on top of the backend-specific types. That would allow me to avoid large changes in bevy_seedling for the time being.

In other words, the simplified API would be immediately useful to me, though I expect to eventually have no need for it.

So if you think it's good for Firewheel overall, like for non-Bevy users who just want a simple way to set the devices, that's great!

[dusky mirage]

Ok, cool! I'll work on that in a bit.

[slate scarab]

A name and ID would be perfect -- that's definitely all I need at the moment.

[steep dove]

to elaborate on this a bit: from my recollection of the picking upstream is that moving large quantities of new code into the repo is just politically hard -- it goes around our normal review practices. but, imo, this is kinda silly. code that gets added to the repo but isn't integrated is harmless, and can be easily removed. but it's way way easier to start with integration once it's there.

here's a pie-in-the-sky proposal.

1. move seedling into the repo as is, keep it named bevy_seedling, keep the existing audio crate, modify absolutely nothing.
2. take the existing audio examples, add a feature flag that makes them use seedling instead
3. maybe add a new example to show off any extra cool features seedling has (use the same flag to switch between the old and new crate)
4. get maintainer approval to move forward with replacing the old crate
5. rename bevy_seedling to bevy_audio, remove the feature flags from the old crate, write an extensive migration guide.

now that's 4 different main-repo prs, seems pretty approachable, and any existing contributor can try out the new stuff out or help

[celest whale]

I'm in favor of this. Note that step 1 will also include "get CI to let us merge it"

[oak walrus]

I'm afraid 2. will introduce visual complexity and make reading and understanding the examples hard. I guess we'd need to split the implementation into two and make the seedling-based ones have the correct flags in Cargo.toml. But overall I agree. This group was created with much smaller scope and without the hindsight that doing an actual audio engine was going to be such monumental task lol. So we kinda need a bespoke strategy to get this work upstream no matter what, IMHO.

[celest whale]

For 2, we could mirror the audio examples too

Even stick them inside of the bevy_seedling/examples folder to start

And add a dev-dep on bevy itself

[slate scarab]

oh ya we could do that

[dusky mirage]

@slate scarab I've added SimpleStreamConfig! https://github.com/BillyDM/Firewheel/pull/97

There's one more thing I want to look in to. I'm not totally sold on using Instant::now in the audio thread. At the very least, it can be disabled if scheduled events are disabled.

[steep dove]

(btw if you are looking for an alternative to Instant::now but still need a clock, https://docs.rs/fast_time/latest/fast_time/ is a neat crate)

[dusky mirage]

Oh, did not know about that. I'll look into it!

[slate scarab]

Although potential imprecision of a few milliseconds may be too great for this purpose.

[dusky mirage]

Ehh, actually it probably doesn't matter it only gets called once every process block, which only occurs about once every 23 milliseconds at the default block size.

And from some testing I did earlier, RealtimeSanitizer doesn't flag Instant::now as being realtime-unsafe.

Though I realized I can avoid using it altogether in RtAudio (because it has a functioning clock built in), so I'll fix that real quick.

Oh wait, nevermind, I can't avoid Instant::now in RtAudio either.

Ok yeah, it's probably good as it is. @slate scarab Do you want me to merge the RtAudio branch into main now so you can work on a Bevy 0.18 PR?

I'll just go ahead and merge it. I'm pretty happy with the API.

@celest whale Oh yeah, it would also be nice if bevy_platform was stabilized before publishing the new version of Firewheel.

[celest whale]

I think that's unlikely to be done in the next couple of weeks, unless @stable berry has very different plans, so don't wait on us this time

My hope was that we could start this with the 0.19 release

[dusky mirage]

Ok, that's fine!

[stable berry]

Yeah stabilization is a big step and there are a number of unresolved problems. We might be able to swing it in 0.19, but it’s not my immediate priority

[dusky mirage]

@slate scarab Is this PR good to you, or is there more that needs to be done for Bevy 0.18 support? (I know I also need to update the rtgc dependency, which I'll do after this PR.)

https://github.com/BillyDM/Firewheel/pull/99

[slate scarab]

I believe so, but I can confirm for sure in a few hours.

[slate scarab]

Ah, @dusky mirage, would the "default" status of an audio device fall under the additional information that an Enumerator would provide? We actually depend on that on bevy_seedling for stream management to some extent.

No worries if that's the case. If so I'll probably need to grab that additional information in the end!

[dusky mirage]

What do you mean by "default status"? Like which audio device is the default one?

[slate scarab]

Yes, that’s what I mean.

[dusky mirage]

Yeah, that's information that the enumerator provides.

Both CPAL and RtAudio have different ways to get the "default device" unfortunately.

[slate scarab]

Ah that’s too bad

Well, I may have to accelerate my plans a bit.

[dusky mirage]

But if you want to use the backend-agnostic input_devices_simple and output_devices_simple methods, then I made it to where the default device is the first one in the list. https://github.com/BillyDM/Firewheel/blob/30f462158da474d7e75d6edfcc07d09cc12f183c/crates/firewheel-graph/src/backend.rs#L37

[slate scarab]

Ah, that should be enough then.

[dusky mirage]

Do you think it would be better if instead DeviceInfoSimple had a is_default_device field?

[slate scarab]

I shouldn’t need it — I shuffle the data into a different struct anyway.

I can’t say whether it’s better though. Possibly

[dusky mirage]

Ok, so the "first one in the list" is a better API you think?

Ok

I guess from a UX design point, it would make sense that the default device should be the first one in the dropdown list.

[slate scarab]

I don’t mind the ordering-as-default, yes. I think that’s reasonable.

[slate scarab]

Ah, by the way this looks sufficient. Although if you don't mind, we could also bump the bevy_platform version.

[dusky mirage]

@slate scarab I just realized that firewheel-macros depends on bevy_macro_utils, which in turn is depended on by firewheel-core. It would be nice to remove this bevy dependency for when bevy_platform stabilizes (then we can stabilize firewheel-core!).

[dusky mirage]

bevy_platform is defined as a workspace dependency, and I updated rtgc, so everything should be up to date now!

[steep dove]

Does audio use any of the realtime scheduling stuff on linux? SCHED_FIFO and similar?

[slate scarab]

I can’t say myself — that would be at the level of a library like Alsa I believe.

[celest whale]

@rapid umbra reminds me that version ranges exist

Using those for bevy_platform etc for this crate feels totally fine

[rapid umbra]

@fossil anvil explained that if any of the dependencies using range versions are duplicated, it will fail to compile, instead of "namespacing" each duplicate

[fossil anvil]

note that, however, it's OK to use range dependencies for private dependencies, that is, ones where what you depend on doesn't affect any types or traits in your public API

so for example if you depend on itertools, ranges can reduce the chances of building two versions of itertools, with no disadvantage as long as none of your public functions visibly return an iterator type from itertools

[dusky mirage]

The RtAudio backend supports realtime scheduling. You can set it in the stream config.

@slate scarab Do you think Firewheel 0.18 is ready to be published on your end? And if so, should I mark it as a beta release or a full release?

(Oh yeah, and don't forget to update the symphonium dependency version for bevy_seedling!)

[slate scarab]

Yes I believe so. I think a full release should be fine! I’ve tested everything on 0.18 (minus these last commits). Although if you want a final confirmation, I can update everything in about 45 minutes.

The glam 31 feature is great —we’ll need it for 0.19 I believe.

[dusky mirage]

Ok yeah, I'll wait for a final conformation!

[steep dove]

how does this work? looking into it, it seems like you need to run rt processes as root?

is it that alsa/jack/pulse whatever run in a separate privileged rt process and you just sort of send it stuff?

[dusky mirage]

Look at the priority field for the config. https://docs.rs/rtaudio/latest/rtaudio/struct.StreamConfig.html

Oh, I should probably clarify that the priority is a number between 0 and 100.

But keeping the default priority of -1 is probably fine. The most important thing is to set that realtime flag.

https://docs.rs/rtaudio/latest/rtaudio/struct.StreamFlags.html#associatedconstant.SCHEDULE_REALTIME

[steep dove]

right but dosn't linux ignore rr scheduling unless you are running as root or have a granted capability?

sched_setscheduler just doesn't do anything for most processes

[dusky mirage]

Correct. It's pretty typical for musicians to install the realtime-priveleges package and add their user to the realtime group. https://wiki.archlinux.org/title/Realtime_process_management

[steep dove]

ah, gotcha. i was looking into using realtime scheduling for some non-audio bevy stuff (the linux scheduler hates this one weird trick) but that's not something most people are going to be willing to do.

this isn't something we would ever expect a normal bevy game app to be relying on, basically? right?

[celest whale]

Are you mining crypto on the audio thread???

[steep dove]

all i want is nanosecond-accurate thread wakeups

is that too much to ask

(yes, it is, it very much is)

basically, turns out linux really hates when you do thread::yield_now in a hot loop, unless you're doing rt stuff. i was doing this because macos loves it, had to figure out something else for linux.

[slate scarab]

Okay it all looks good to me.

[dusky mirage]

Alright, Firewheel 0.10.0 is published! 🎉

[slate scarab]

oki a new bevy_seedling version is published too

[rapid hedge]

@slate scarab about to update bevy_steam_audio to the newest bevy

anything I should know about? 🙂

(using the newest seedling and firewheel too ofc)

[slate scarab]

i think there haven't been any major breaking changes since 0.6

it should be pretty easy!

[rapid hedge]

neat!

[rapid hedge]

update: already done

you were right

[sturdy prawn]

y'all should probably add bevy_seedling and bevy_steam_audio to Bevy Assets
https://bevy.org/assets/#audio

@slate scarab @rapid hedge (though corvy saw already :3)

[hushed widget]

(and make a nice image for it)

[celest whale]

🌱

[sturdy prawn]

someone get this crow to 100 stars

[slate scarab]

i have been peeking at it xD

[vocal dock]

Done

[slate scarab]

Okay I finally made some time to write up a bit about the current state of the working group.

https://hackmd.io/@V7X4pNgkQNyPmlDV9LVOUQ/BJjx7Zj8-g

@dusky mirage @oak walrus and any others, feel free to make corrections and suggestions! I think it captures the overall thinking of the group pretty well, but I may have overlooked some details.

This is intended to

1. Indicate where we're at (this should help orient new contributors and communicate the advantages of our work)
2. Shed some light on the technical decisions we've made
3. Provide a brief, high-level overview of the overall architecture

[slate scarab]

It does not make any plans or suggestions about upstreaming, as I think there's still a bit to discuss there, particularly with the upcoming project goals.

[celest whale]

Yep, I would love to see a list of things you would like to do before / after upstreaming and some rationale for that 🙂

This is excellent writing though, and a really good basis for release notes

[steep timber]

As a lurker without any particular game audio experience, I found that doc super informative

It makes me want to try out bevy_seedling

[oak walrus]

Yeah, I'm going to join the praise here, this is a great writeup, thanks for the shout out I didn't expect to be mentioned haha

It does feel good to see your vision take shape when you had an idea of what to do but not enough time to carry it. Not trying to take all the credit here, but I do love this little micro-community that spiraled into its own passionate group, and I can't wait for Bevy to have a proper first-party audio engine!

[dusky mirage]

Sorry, been busy with other things. I'll get to this tomorrow!

[real mica]

This is extremely well written. Nice job!

[dusky mirage]

Looks good!

One thing you might add to the section "High-performance, real-time safe processing" is a technical explanation of how Firewheel is able to achieve those performance numbers:

• Block-based processing architecture - While ⁨rodio⁩'s per-sample iterator processing architecture is easier for users to understand, block-based processing allows for much greater cache efficiency, much more aggressive compiler auto-vectorization, more efficient ⁨memcpy⁩ operations, and far less bookkeeping overhead (you only need to do bookkeeping once per block of samples instead of once per sample). Back in 2024 ⁨kira⁩ also used per-sample processing, though it has since switched to a block-based processing architecture.
• Silence flag optimizations - Inspired by the CLAP audio plugin API, Firewheel's nodes have access to optional hints as to which audio channels contain silence, allowing nodes to choose to bypass processing altogether if the input is silent. Especially useful when working with pools of nodes where most of the time the majority of the nodes have no work to do.
• Specialized processing loops - Some common nodes have multiple hand-written processing loops, each optimized for a different state.
• Event-based state synchronization - All nodes can use Firewheel's built-in event/diffing-patching system for synchronization state between the game thread and the audio thread. This event system uses realtime-safe message channels.
• Realtime garbage collector - Nodes have access to a realtime-safe garbage collector, which ensures that all heap-allocated data gets dropped in a realtime-safe way.

Yeah, and the main blocker to streaming is figuring out how such a system should even be integrated. Like how will it integrate with Bevy's asset system? And how will it work in webaudio? Do we just disable that feature for web?

[slate scarab]

Right, that to me seems like a post-upstreaming step due to such questions.

If we're lucky, the recent workin #ecs-dev may help (assets as entities).

If we're not, well other parts of the engine really need asset streaming as well, so it's not just audio that would benefit.

[dusky mirage]

And it would be easier if we limit ourselves to only supporting streaming for audio file types which play nicely with the concept (namely wav, ogg, and opus). File types like mp3 are notorious for not behaving well with streaming (well you can stream it, but there is no way to seek to a different playhead position without decoding the whole file first).

But in theory it should be possible to create a streaming ⁨SampleResource⁩ that has a handle which collects samples from any thread and then sends them to the ⁨SampleResource⁩ in a similar manner to my ⁨creek⁩ crate.

That would also require me to add a method to ⁨SampleResource⁩ to tell the resource to preemptively cache samples before a loop (that will be easy to add though.)

[compact sphinx]

i could imagine that a web application may want to stream audio from a remote source - maybe that's a file on a disk somewhere but maybe it's proximity chat? could the streaming API support that as well? i.e. be agnostic wrt whether the streaming audio is from a local file or from the network, or from a local microphone

[dusky mirage]

Or a simpler, more foolproof approach would be just to have a separate ⁨StreamingSampler⁩ node, one that doesn't support special effects like pitch shifting.

Because trying to get those effects to work seamlessly with a standard⁨SampleResource⁩ could be challenging.

[dusky mirage]

Or actually, maybe it won't be that challenging. The user will just have to be aware that they can't play a sample too fast or the streaming can't keep up.

[slate scarab]

ya i think that's totally okay

a caveat of "streaming can't go too fast (or maybe jump around too much)" but still otherwise fitting in the same shape as non-streamed audio will make it very easy to use

[dusky mirage]

But yeah, doing the math, a long 10 minute decoded stereo audio file is about 230MB of memory. Not ideal.

(10min * 60sec * 48000samples-per-sec * 2channels * 4bytes-per-sample)

In theory we could halve that by dithering it down to 2 bytes-per-sample, but obviously streaming is the better solution.

As for documentation, I could definitely use some help with that. 😅

[slate scarab]

writing documentation is a painful exercise xD

i could see that happening post or even during the upstreaming process

[dusky mirage]

Yeah, one thing I learned from Firewheel is that I really need to reduce the amount of user-facing code I write (not in Firewheel, my other projects). Keeping up with maintenance and documentation for an open source library is a lot of work.

Kind of gets in the way of making a Digital Audio Workstation. 😅

[limpid mason]

Oh, are you building one?

I thought that was explicitly out of scope for Firewheel 😆

[slate scarab]

ya it different

[dusky mirage]

Yeah, it's a separate project. Firewheel was actually partly an excuse to learn how to better structure my DAW audio engine.

And I have learned a lot while creating Firewheel!

Albeit Firewheel did end up being a more DAW-like audio engine that I originally planned.

[limpid mason]

Do you have a link? I'd be interested to take a look

[dusky mirage]

There's not much to show yet, mostly old discarded attempts. I haven't made the new code public yet. https://codeberg.org/Meadowlark/Meadowlark

But I do have a Discord server!

It's been a multi-year long passion project that I hope to make a career out of someday!

[limpid mason]

The logo looks familiar, maybe I was following it already haha

(It's a beautiful logo)

[dusky mirage]

Thanks!

[limpid mason]

Oh yea, I'm already on the Discord. small world

[dusky mirage]

I literally just traced around a picture of a Meadowlark with the pen tool in Inkscape. 😁

[limpid mason]

Hey, if it works... :D

[stable berry]

Better Audio

https://github.com/bevyengine/bevy/issues/23031

Note that I've temporarily put this in the "inactive" state, as while it has a working group (as defined by the old system), no SME is currently staffing it / providing direction. I'd like to spend my time on this / I consider improving Bevy's upstream audio to be a high priority, but I don't currently have the bandwidth to "staff" it.

I think this Working Group should continue to function as it has (relatively autonomously) until someone from leadership can take a more active role.

Added some details on Firewheel and bevy_seedling to the Goal

[faint wigeon]

there's a design doc in the pins here

[stable berry]

Added!

[celest whale]

Important analysis / discussion here

Vero's analysis of viability of upstreaming firewheel in 2026-02 https://hackmd.io/@bevy/rkwFscXObx

[faint wigeon]

i just want to say that regardless of what upstreaming looks like, the r&d this group has done has been absolutely crucial and i think a great model of working together in our community

[dusky mirage]

Bevy will be the best game engine ever made!

[hushed widget]

I hope not, I hope most of the things we do in this age won't be the best ever, that's so depressing to think "ok we've peaked, it's all downhill from there"

[slate scarab]

okay how about the best engine so far :3

[hushed widget]

that I can get behind 😄

[rapid hedge]

@cold isle I'd like to upgrade audionimbus, but I'm having some trouble :/

• SteamAudioContext not being Sync makes it really tricky to actually use it in the ECS and the audio thread. I probably need to put it in a Sync wrapper with an unsafe fn get() or something like that. Note that I cannot use Mutex or RwLock inside the audio thread. Not sure this is something that is actually "fixable" upstream due to Steam Audio's inherently weird threading, just mentioning.
• Simulator having a lifetime makes it impossible to store inside the ECS (this is a hard wall rn)
• Same for Source
• ReflectionEffect having a generic means that I need to propagate the generic to literally every top-level method, which is not really compatible with the rest of my API. Could we have an enum instead? If not, I'll probably need to hardcode Convolution as a bandaid
• Maybe some others too, hard to tell due to how many compile errors I have

There's some other types that are now no longer Sync or Clone, but those don't need to be on the audio thread, so I'm fine with Mutex and Arc for them

[rapid hedge]

Especially the lifetimes currently make it impossible to upgrade rn, so I'll have to stay on audionimbus 0.11 for now

I see that these changes are for memory safety, so thanks a bunch for that

just needs some tweaking 🙂

[cold isle]

I'll have more time this weekend, happy to look into it 🙂
I suspect the main roadblock is going to be the types not being Sync anymore due to SA. Quick question though: why do you need to access it in the audio thread? I suppose it's for the effects? Can't you create the effects outside the audio thread and send them there instead? (wild guessing since I'm not familiar with your API)
Regarding Simulator: I need a way to keep the devices alive for the duration of the simulator, but if lifetimes are the problem I can probably work around them.

In any case, I'll take a look this we

[slate scarab]

Doesn't steam audio do this itself? For example, I believe it's being passed around in the audio thread here: https://github.com/ValveSoftware/steam-audio/blob/5b81be1b06d290dbe6f6e480a2c2404010cb28eb/unity/src/native/spatialize_effect.cpp#L787-L967

If I'm not mistaken, the gContext variable is just a global that owns the context.

Not sure where it's declared, but there is an extern here.

[cold isle]

Doesn't steam audio do this itself?
Sorry which this do you mean? Passing the context around? Yes that's a valid use case because the context is used by both the simulator and the effects (which are typically run on separate threads). It's referenced counted by SA

[rapid hedge]

Wonderful!
yep, basically everything needs access to the SteamAudioContext, so I absolutely need (lock-free!) access to it. I don't think we get around unsafe there. But that's fine, the current code using 0.11 de facto already uses those unsafe invariants

so it's at least not worse than what we have rn

[rapid hedge]

everything else that is no longer sync doesn't need to be on the audio thread, so Mutex is fine

[slate scarab]

I guess what I mean is that it appears they use it as if it's Sync (it's placed in a global after all for the Unity integration). Though they do a lot of naughty things so that may not mean anything.

[rapid hedge]

judging by the rest of the API, SA in general is not really thinking in terms of "is this struct Sync" but instead "is this specific method safe to use in multiple threads"

so the Syncness of many types is situational

which Rust does not like at all

-# It's also really messy

[rapid hedge]

I can imagine gContext being the same way: not really Sync, but if you behave as dictated by the documentation regarding method calls, you're fine

[river marsh]

Ah, didnt know this group existed. I'm gonna do some more passes on firewheel tonight

[rapid umbra]

What are the most relevant issues/PRs/Goals for this working group at the moment?

[rapid hedge]

@cold isle anything I can do to help?

I think as soon as those lifetimes are gone, I can at least bandaid the other things

it's only the lifetimes that are a hard blocker

[slate scarab]

does the simulator need to stick around between frames? or can it be made static in some circumstances?

[cold isle]

Can you guys take a look at this PR when you get a chance? https://github.com/janhohenheim/bevy_steam_audio/pull/53
I made a first pass using 'static as @slate scarab suggested, seems to work fine
But maybe there's something else I'm missing which makes using lifetimes a hard blocker?

If everything is OK, I'll roll the update tomorrow/later this week

[rapid hedge]

Oooh smart!

[rapid hedge]

that looks very good to me 🙂

I can't easily verify that the code this calls on your end doesn't allocate or use any locks inside the nodes

but I trust you made sure of that

@slate scarab would you mind also taking a look?

also @cold isle is the version of AN this is referencing already on github?

If so, I would like to try patching to it in my other project and see if everything still works

[slate scarab]

Yes I will take a look today

[slate scarab]

oh, yes i'd like to know what version it is so i can skip around between them

[cold isle]

Yep, the PR points to a local version of AN pending the release, it's the same version as the master branch that's on GH

[cold isle]

I published the new version (v0.13.0) and updated the PR, that'll make testing easier 🙂

[cold isle]

Right now there is a concern with nodes because they all call get_outputs(), which is not safe to call while a simulation is running (see https://github.com/ValveSoftware/steam-audio/issues/256). This was fixed with a lock in https://github.com/MaxenceMaire/audionimbus/pull/46 to prevent race conditions. Whether it's a big contention point - I'm not sure.
But it's more of an architecture problem; that should be easily fixable in a follow-up though. The best way would be to shift get_outputs() to the simulation block and use a triple buffer that the nodes can peek into, instead of each individually calling get_outputs().

[rapid hedge]

Yeah we can’t have any locks in there, OS is allowed to park the thread then

[rapid hedge]

Yep that sounds like a good strategy, good idea

[rapid hedge]

@cold isle should I wait with merging for this? Or should I implement it myself? 🙂

[cold isle]

As you wish, go with what's most convenient 🙂

[slate scarab]

locks aren’t great, but it’s not literally game over

(i can review now with the info about the version)

[slate scarab]

well, unless the main-thread locking is long-lived

also on Wasm it might cause the main thread to crash

that is, it'll be problematic if the simulator needs to be locked when the simulation is running, since presumably that takes several milliseconds

[slate scarab]

That method also allocates to collect the locks and for the simulation outputs, so it's definitely not good for the audio thread!

[slate scarab]

Oh btw @cold isle, sorry if it's already been noted, but is there some context on why the air_absorption function segfaults when using a callback?

[slate scarab]

Okay, I also think it's good for a merge, with the caveat that we absolutely need to move get_outputs off the audio thread. (Note that we really needed to do this anyway.)

I don't recall if we avoided it because it was annoying or for other reasons (like the outputs not being Send or something).

[cold isle]

Steam Audio was dereferencing a null pointer: https://github.com/ValveSoftware/steam-audio/pull/521/changes
The fix has been merged, so it'll be safe on the next release 🙂

[dusky mirage]

@celest whale @slate scarab @oak walrus @river marsh
I did also think of a different possible direction we could take I'd like to hear opinions on.

Instead of upstreaming all of Firewheel completely, it might be possible to instead have firewheel-core not depend on bevy at all by having a generic type like this:

pub trait CoreTypes {
    type NodeID: Into<thunderdome::ID>;
    type ParamData: Debug + Clone + Send + 'static,
    type MusicalTransport: MusicalTransport + Send + 'static,
    type EventInstant: EventInstant<Self::MusicalTransport> + Copy + Clone + Send + 'static,
    
    // ... and whatever other core types we need
}

pub trait MusicalTransport {
    // transport stuff
}

pub trait EventInstant<M: MusicalTransport> {
    fn to_samples(&self, proc_info: &ProcInfo<M>) -> i64;
}

// ... and whatever other traits we need

This does mean we would have to recreate all of Firewheel's nodes in the bevy repo, but it could be a worthwhile tradeoff.

Of course if you think it would be easier just to upstream all of Firewheel into bevy, that's fine. I'm just trying to figure out the best way to have non-bevy users to easily have access to Firewheel as well.

Hmm, though I suppose we would still miss out on some bevy_platform stuff, but we can make Firewheel use its own non-std stuff.

[slate scarab]

One thing to note is that we can still make it accessible to non-bevy users even if it resides within the Bevy repo. bevy_reflect is an example.

[dusky mirage]

Ah yeah, that could be a smarter route. My only concern is if bevy wanted to go CPAL-only, that would prevent non-bevy users from using a different backend. Though I suppose we could just have a bevy_cpal crate that is separate from the rest of the bevy audio crate.

Oh yeah, and how does the upstreaming process work exactly?

Should I create a bevy_upstream branch in Firewheel, do all of the crate consolidation and renaming there, and then we copy-paste that code into a fork of Bevy in a PR?

[river marsh]

we are not going cpal only

[dusky mirage]

And speaking of names, will the name of the crate just be bevy_audio? I suppose we could also just call it, bevy_firewheel, but that would break the descriptive naming convention of bevy's crates.

[river marsh]

imo lets do more clean-up in repo first, like the stuff regarding microphone silence thresholds etc

bevy_seedling will be the new bevy_audio imo

bevy_firewheel can be bevy_audio_graph

[cerulean shoal]

We do have solari already, but yeah in general name tend to be boring

[river marsh]

i asked cart to reserve that crate

[dusky mirage]

Ah ok.

[river marsh]

i also asked him to reserve bevy_firewheel in case we want to keep the branding

the branding decision is not something i have an opinion on

[dusky mirage]

Yeah, I'm not sure where I stand with the name.

[river marsh]

at the very least it would look ugly to have a non bevy_ prefix crate in crates/

[dusky mirage]

Once this is all upstreamed, my plan is to turn the Firewheel repository just a showcase on how to use the "new bevy audio" crate in a non-bevy project.

Either that, or I might archive the Firewheel repository and make a firewheel-examples repository or something. Actually yeah, now that I think about it, that would make it clearer to users that they shouldn't post issues to that repository.

But if I'm planning to archive the repository anyway, then I'm not so attached to the name anymore.

Oh yeah, I also need to do this: https://github.com/BillyDM/Firewheel/issues/91

[dusky mirage]

rubato also got a big update, so I'll work on that too.

[river marsh]

https://github.com/BillyDM/Firewheel/pull/108

https://github.com/BillyDM/Firewheel/pull/109

https://github.com/BillyDM/Firewheel/pull/110

[river marsh]

https://github.com/BillyDM/Firewheel/pull/111

[river marsh]

https://github.com/BillyDM/Firewheel/pull/112

[river marsh]

https://github.com/BillyDM/Firewheel/pull/113

[slate scarab]

don’t process twice for no reason

[river marsh]

https://github.com/BillyDM/Firewheel/pull/114

@dusky mirage i need reviews

im starting to get to the part where PRs are stepping on each others toes

would like to get this stuff merged and then continue

[dusky mirage]

Ok, I'll get to it in a bit!

[river marsh]

ty!

[hearty bear]

follow up pr from "dont normalize twice"?

[dusky mirage]

@river marsh Looks good! I just have a small question/request for https://github.com/BillyDM/Firewheel/pull/114 and https://github.com/BillyDM/Firewheel/pull/110.

[river marsh]

whats the request for the first one?

[dusky mirage]

I meant that the first one I had a question, and the second one I had a request.

[river marsh]

ohh

ask away

[dusky mirage]

Is the #[inline(never)] required for the #[cold] thing to work?

[river marsh]

yeah, if the code gets inlined then the hint is lost

you can omit the inline(never) but then the compiler may inline it at higher optimization levels

thats my understanding at least

realistically the only true answer to this question is benchmarks

thanks for reviewing everything

we're getting closer to upstream i can feel it

[river marsh]

also i think the cargo tomls already have appropriate disclaimers

i just wanted to note it in the code really

[dusky mirage]

Oh right, I forgot I did that. 😅

[river marsh]

@dusky mirage one more https://github.com/BillyDM/Firewheel/pull/116

[dusky mirage]

Turns out rubato made a significant change to its API. It's fine, but it does mean updating fixed_resample is taking a bit longer than expected.

[inland hollow]

Is it appropriate to ask questions here as an outsider? If so...
Will the new audio implementation support fractional resampling? The current AudioSource is very clicky when trying to push 48000 samples/sec through it (need to use that sampling rate because Opus)

[celest whale]

This is very helpful

Requirement gathering is important!

[slate scarab]

Yeah, the streaming nodes support this via rubato.

[steep dove]

and, for future reference, these channels are more about "what we talk about" than "who you are". everyone is welcome to participate.

[inland hollow]

I see, thanks!

[slate scarab]

Oh, I guess to be more precise we're using BillyDM's fixed_resample crate, where rubato is optional.

[inland hollow]

hasenbanck's resampler proved useful for me (it was easy to implement)

[toxic ingot]

Heya, I'm trying to implement a clap plugin audio node for Firewheel and I'm a bit stuck. Here's how I best understand the situation:

Firewheel has the concept of a "Node" on the game thread and on an audio thread, both traits:

• Game thread: AudioNode
• Audio thread: AudioNodeProcessor ['static + Send]

The type implementing AudioNodeProcessor must be constructed from the type implementing AudioNode and sent to the audio thread, thus the Send constraint.

On the CLAP side, we have more traits:

• Compile time only: HostHandlers ['static]
• Main thread (Game thread?): MainThreadHandler
• Shared between threads: SharedHandler [Send + Sync]
• Audio thread: AudioProcessorHandler [Send]

The types implementing these traits are used for communicating between host (the firewheel node) and clap plugin.

For actually instantiating the plugin, we have three? types:

• Main thread? (Game thread?): PluginInstance [explicitly !Send]
• Audio thread?: StartedPluginAudioProcessor [explicitly !Sync]
• Audio thread?: StoppedPluginAudioProcessor [explicitly !Sync]

It's my understanding that PluginInstance is meant to stay on the main thread, and the AudioProcessor types are meant to be sent to the audio thread. If that's the case, then it seems like a pretty direct analogue to the AudioNode and AudioNodeProcessor types in Firewheel.

construct_processor, however, takes a immutable reference to self, and activating a plugin using a PluginInstance requires a mutable self. PluginInstance::activate and StoppedPluginAudioProcessor::start_processing are also fallible, which can fail not just from using the API wrong, but also if the plugin fails in some way.

Would it make sense to make AudioNodeProcessor::process take a mutable reference and return Result? Up until now the construction of any AudioNodeProcessors didn't mutate the AudioNode they were constructing from and could never fail.

Also, PluginInstance isn't POD, and so doesn't work with any of the bevy or serde features. In which case, it may make more sense to put both the PluginInstance and the StartedPluginAudioProcessor on the AudioNodeProcessor instead. However, PluginInstance is explicitly !Send and the AudioNodeProcessor trait is explicitly Send (I'm assuming so that the processors can be sent between different audio threads?), making them incompatible. The previous try at CLAP plugins got around this with unsafe impl Send for ClapPluginProcessor {}, but that doesn't seem like a good solution..

Maybe there needs to be a global database of loaded Clap plugins that lives on the main thread somewhere that AudioNodeProcessors can access during instantiation? Does such a place exist?

[toxic ingot]

Is something like this crazy?

struct ClapPluginRegistry {
    plugins: HashMap<String, PluginInstance<FirewheelClapHost>>
}

impl ClapPluginRegistry {
    fn new() -> Self {
        Self {
            plugins: HashMap::new(),
        }
    }

    fn retrieve_plugin(&mut self, id: impl AsRef<str>) -> Option<&mut PluginInstance<FirewheelClapHost>> {
        self.plugins.get_mut(id)
    }

    // etc
}

lazy_static! {
    static ref CLAP_PLUGIN_REGISTRY: ThreadBound<RefCell<ClapPluginRegistry>> = {
        ThreadBound::new(RefCell::new(ClapPluginRegistry::new()))
    };
}

[slate scarab]

Hm, I haven't taken the time to fully understand the problem, but have you seen the custom_state method on AudioNodeInfo? https://docs.rs/firewheel/latest/firewheel/node/struct.AudioNodeInfo.html#method.custom_state

The AudioNode::info is called once, so you should be able to stuff !Send data in there.

You can then access it on the context's thread via node_state https://docs.rs/firewheel/latest/firewheel/struct.FirewheelCtx.html#method.node_state

(This is where the PluginInstance would go.)

[toxic ingot]

hmm ok thanks I'll take a look

[toxic ingot]

Yeah that does fix the issue of where to put the PluginInstance, but there's a lot of places where Clap can fail.

To keep the AudioNode type component/reflect/etc compatible, none of the clap types can be fields of that type, that means that all the operations for taking a path and an ID and extracting a PluginInstance from it that can be used would need to be in AudioNode::info, which is currently infallible. For example, it can fail loading the dynamic library, getting the plugin factory, finding a plugin of the particular ID we're interested in, etc.

AudioNode::construct_processor also is currently infallible, but both activating the PluginInstance and starting processing are fallible operations.

We don't want to panic if the user loads a plugin that can't process for some reason or if they provide an ID that doesn't exist in the clap bundle for example (as much as we can help, as far as I understand loading a dynamic library at all is opening yourself up to a can of worms)

Would it be reasonable to make both AudioNode::info and AudioNode::construct_processor fallible?

[slate scarab]

I've actually wanted some sort of fallibility for a while myself.

The question of course is what to do with the error type.

[toxic ingot]

hmm yeah

[slate scarab]

I'm not totally sure I like it as an associated type on the trait.

Sounds like a generics proliferation nightmare for little gain.

An anyhow-like type is probably fine.

[toxic ingot]

Can you do something like

fn construct_processor(
        &self,
        configuration: &Self::Configuration,
        cx: ConstructProcessorContext,
    ) -> Result<impl AudioNodeProcessor, impl Error>;

I'm still kinda wrapping my head around traits and generics and such

[slate scarab]

Rust won't be happy with impl Error

[toxic ingot]

It's ok with -> impl AudioNodeProcessor though?

what's the difference

[slate scarab]

While impl types are opaque, they still need to resolve to a single type. Generally, you always return a single AudioNodeProcessor, so there's no issue. But you may have multiple error types within a single function. Rust won't be able to convert these into impl Error with ?.

[toxic ingot]

If you make a new error type with thiserror or something and use [#from] would that work?

Cause it's one type

[slate scarab]

I'd recommend something like

struct NodeError(Box<dyn Error>);

impl From<T: Error> for NodeError {
    fn from(value: T) -> Self {
        Self(Box::new(value))
    }
}

[toxic ingot]

This is similar to what anyhow does yeah?

[slate scarab]

In terms of API surface, but anyhow uses some tricks to make the error one pointer on the stack instead of two.

[toxic ingot]

except without wide pointer trickery yeah

[slate scarab]

But we don't need to worry about that here -- node creation will never be on any hot path.

[toxic ingot]

theoretically, just so I understand, if we had just one error type for each implementation of AudioNode (and used thiserror for the ? conversion to specific variants of the error), then would this be valid?

I'll test

[slate scarab]

No it'll be an issue with type inference using ?. Rust won't do any kind of full-program analysis.

[toxic ingot]

I see

[toxic ingot]

Do you think I should split the error stuff into a PR first or should I wait? It seems like it will be useful to you and I'm not yet sure if this CLAP stuff will work out

[slate scarab]

Yeah I'd be interested to see what BillyDM thinks. But I have certainly had one or two fallible nodes that will currently panic in edge cases.

[toxic ingot]

panic during processing or during initalization like info and construct_processor?

[slate scarab]

construction

[toxic ingot]

gotcha

would there be any other methods you'd want to see fallible?

[slate scarab]

Besides those two? I think they represent the main API surface for node authors, so that seems like a good PR scope.

[toxic ingot]

do you remember the nodes that panic atm? I can clean that up as part of the PR

or how to search for them

is it just an unwrap or expect?

I guess there aren't that many haha

[slate scarab]

I believe they are not a part of the firewheel repo.

[toxic ingot]

ah I see