Better Audio

[rapid hedge]

you know what

let me reboot

[slate scarab]

and uh... maybe turn down the volume a bit

[rapid hedge]

why not

[limpid mason]

You're on Linux right?

[limpid mason]

I can offer Pop_OS

Wait, am I crazy or did the green used in cargos logs change?

It doesn't match the green of the info logs anymore 🤔
(I guess the app not working might be a more pressing issue though :D)

[rapid hedge]

Welp

[rapid hedge]

hehehehe

Here's my minimal example: https://github.com/janhohenheim/test

hhh@hhh-fedora ~/g/test (main)> cargo --version
cargo 1.92.0-nightly (966f94733 2025-09-16)
hhh@hhh-fedora ~/g/test (main)> bevy --version
bevy 0.1.0-alpha.2
hhh@hhh-fedora ~/g/test (main)> google-chrome --version
Google Chrome 140.0.7339.207
hhh@hhh-fedora ~/g/test (main)> wasm-bindgen --version
wasm-bindgen 0.2.101

I'm out of ideas

Which means, the worst outcome has come

I have to boot Windows

:[

[slate scarab]

oh no

[limpid mason]

I also got this prompt text duplicated for some reason

In Firefox it loads indefinitely with this error
Is it supposed to work in Firefox? Or does it use webgpu?

[slate scarab]

huh (it seems to work fine for me on macos)

That looks like the URL from the dynamically generated js shim though for firewheel-web-audio.

[rapid hedge]

it does not use webgpu

[slate scarab]

looks like an overly zealous CSP header

[hearty bear]

Works for me on firefox (arch) and chrome (mac)

[slate scarab]

not sure why that would come up in firefox in particular though

[rapid hedge]

;-;

setting up my windows rn

[limpid mason]

Oh wait, is it supposed to just show the spinner indefinitely and just play sound?

Then it's working for me on both Firefox and Chromium
Once I click the page to activate the audio context, I hear the crow

[slate scarab]

My bad, I should have mentioned that. The audio context is not allowed to start without some user input.

[limpid mason]

Yea that makes sense. I was more confused that the loading spinner didn't go away :)

[slate scarab]

Well that should also go away 😅

[limpid mason]

Ok, nevermind :D
That stays for me on both Firefox and Chromium

[rapid hedge]

at least you have audio

[slate scarab]

any errors in the console? (that might help diagnose the stuck spinner)

[limpid mason]

Can we combine to get the full game?

[rapid hedge]

[limpid mason]

I did

bevy run --example one_shot --features web_audio web -U multi-threading

With latest main of bevy_seedling, latest nightly and alpha2 of the CLI

[rapid hedge]

did you also change the example to actually use the new web backend?

(see my test repo)

[limpid mason]

😶

[rapid hedge]

shit

It works on Windows

;-;

Well, "works"

It immediately crashes after the CAW

[hearty bear]

You missed your monthly payment to IBM

[rapid hedge]

but at least it plays audio

[rapid hedge]

The previous fedora lead is on this server

should just ping him lol

[slate scarab]

I would like to make it "just work" without changing the plugin invocation, though. Although it would require completely abstracting the backend. I'm not sure that's ideal in situations where people want more direct access to, say, cpal.

While the current multithreading audio situation in bevy_seedling is better than it was, it's still a touch awkward.

[limpid mason]

Trying that now

[rapid hedge]

On firefox, the spinner also doesn't go away, but there's no crash

[limpid mason]

We're getting there though!
Just a couple of months ago, I feel like multi threaded audio was completely out of reach (in my mind at least)

[rapid hedge]

Just having it be possible at all (well, at least for others ) is magical haha

Oh I have an idea to see if this is a new bug in seedling / firewheel or in chrome

let me boot up foxtrot!

[limpid mason]

not gonna lie, I think at some point I also want trunk's caching of the different wasm-bindgen versions
It's nice that the CLI can re-install it automatically, but it sure is annoying that whenever you change projects you have to do it :D

[rapid hedge]

@slate scarab this error does not happen on the same Chrome version when playing https://janhohenheim.itch.io/foxtrot, which uses the audio web backend in Bevy 0.16

Gonna be AFK for a while and then again bash my head against my keyboard

but on linux

[limpid mason]

Ok so on Jan's reproduction it works on Firefox and on Chromium the audio plays but I get this error:

No idea how up-to-date my Chromium is though

[limpid mason]

So it's the same error that Jan got on Windows

[drowsy dome]

test~

[rapid hedge]

test received

[drowsy dome]

it seems like unreachable code is being reached

so uh

idk what h79ac3a2b468bf3b8 is tbh

dont quite understand why nothing is mangled except for those. surely it's not a closure

[rapid hedge]

@slate scarab I just downloaded a fresh chromium and would you look at that, it works

It still crashes with that error after a few seconds

But it plays audio!

Here's an issue: https://github.com/CorvusPrudens/bevy_seedling/issues/60

[rapid hedge]

And here https://github.com/CorvusPrudens/bevy_seedling/issues/62

[rapid hedge]

this one for sure is blocking for me :/

so I would be glad if someone could confirm whether they can reproduce it

[rapid hedge]

Alright, Foxtrot main is now on 0.17.0-rc.

Again, I can't publish it on itch.io because firewheel panics on web (issues above)

but this is a start 🙂

[limber kernel]

Running fine for me under macos chromium stable, let me try your specific chromium version

or is this gonna probably be linux specific?

[rapid hedge]

could be

[rapid hedge]

thanks for checking 🙂

[slate scarab]

Also, are we sure it’s seedling that panics here? This would be a huge pain — I don’t even know where to start since I can’t reproduce it and the panic message is... less than useful.

[slate scarab]

(in 60, not 62)

[rapid hedge]

Hmm, I can play around with the code to see if I can figure something out.

[slate scarab]

Is the terse panic message due to symbol stripping or other optimizations maybe?

[rapid hedge]

Can you reproduce 62?

[slate scarab]

I haven't had any luck, but at least there's an obvious patch (just clamp the duration so it's positive 😅).

[rapid hedge]

I don’t think I have any enabled for the reproduction. But it kinda has to, right??

[rapid hedge]

Hmm, 60 is reproducible for others, wonder why it isn’t for you

[slate scarab]

Right yeah that part means it's not just spurious

[rapid hedge]

I can patch the app later to a hotfixed version of firewheel

And see if that helps

Or if this is a symptom of something else that’s wrong

[slate scarab]

Although (and sorry to harp on this) have we actually established 60 is related to seedling or firewheel? Sorry if I missed any context on this.

[rapid hedge]

Fair!

[rapid hedge]

It only happens when I change the minimal repro to use the new backend

And does not happen for my old foxtrot build

[slate scarab]

dangit

[rapid hedge]

So something broke in the meantime

I should also try to compile a fresh 0.16 version of the code that uses the new audio backend

If that also fails, it’s probably related to either rustc or wasm-bindgen

If that does work, then it has to be firewheel or seedling

[slate scarab]

What's particularly odd is that there's only one message. Is there a "text decoder not available" error anywhere in proximity?

Normally, a panic produces exactly two messages.

[rapid hedge]

I can check later, but I don’t think so

[slate scarab]

A panic in the audio processor itself can't get its message out, so that can produce just one (along with a text decoder not available message).

I've never seen just one message otherwise though, which could indicate some interaction deeper in the stack. Which would be bad.

[rapid hedge]

Maybe some important new feature is not enabled?

I copy-pasted the bevy features from the seedling docs

But didn’t question them

[slate scarab]

tbh I copy/pasted those from the default feature set of bevy

They should be solid.

[rapid hedge]

Ye

[rapid hedge]

This is probably the best lead

I hope for you it’s rustc weirdness lol

Then we can just isolate the latest working version and be happy

And write a bug report I guess

[slate scarab]

Okay for 62 would you be able to patch firewheel-web-audio with rev = "65888f3" (dur-fix branch) @rapid hedge?

The only possibility is that the browser is giving us bad numbers, so there's really nothing we can do there but clamp it to zero.

[rapid hedge]

Okay on it now, compiling 0.16

@slate scarab good news and bad news!

0.16 gets the same error

BUT

If I replace the minimalplugins + something blurb at the beginning with DefaultPlugins, I get an actual error!

guess which one lol

So my two crashes are actually the same 😄

[slate scarab]

ah, i guess it was missing the log plugin

[rapid hedge]

it was not

which is weird, right?

[slate scarab]

wtf

[rapid hedge]

some plugin in there is doing some wiring that we don't know about

[slate scarab]

Hm, well that’s nice to see at least.

[rapid hedge]

I'll check seedling 0.4 next

that's the one I used for the last foxtrot deployment

@slate scarab I'm sorry to say, seedling 0.4 works 😄

I'll update the issue real quick

then try your patch

[slate scarab]

That would make sense, since the troublesome behavior I just patched in firewheel-web-audio was introduced along with 0.5

Assuming we’re talking about 62.

[rapid hedge]

60 and 62 are the same, so yes

[slate scarab]

Were you able to test the latest seedling with this patch btw? Just clarifying!

[rapid hedge]

I did not test the patch at all yet

I only verified that 0.14 has no error, but 0.15 and 0.16 do

Want to write down my findings in the issue first before I forget 🙂

https://github.com/CorvusPrudens/bevy_seedling/issues/60

there

now let me try the patch

Using

[patch.crates-io]
firewheel-web-audio = { git = "https://github.com/CorvusPrudens/firewheel-web-audio", rev = "65888f3" }

[slate scarab]

hm, it should be in there with the web audio feature

[rapid hedge]

Cargo tree says it is

oh lol

cargo update

now it accepts it

let's compile

No crash in the minimal example 🎉

let's try foxtrot now

Okay, works too

I do get these though

Don't know if they're from seedling or somewhere else

it doesn't crash the app

[slate scarab]

Now this could be papering over a more serious issue. It shouldn’t be possible that the browser delivers us timestamps that would produce a negative duration in the first place. But since it clearly is doing that, it’s possible the timing information is bad enough that it could interfere with the event system.

I wouldn’t think this is the case — these negative timings should be infrequent and clear themselves up, but I suppose it’s possible.

The result would be potentially inaccurate or, in the worst case, dropped events.

[rapid hedge]

that's when I change states, so a whole bunch of sample players will be despawned

[rapid hedge]

If that is the case, why does it work for seedling 0.4?

[slate scarab]

that looks like cursor lock tbh — is that crazy?

[rapid hedge]

Uhh could be. Changing the state forces a cursor lock change

[slate scarab]

we didn’t attempt to fetch as accurate timings

[rapid hedge]

oh I see

[slate scarab]

I did see more angry messages about cursor lock when I was testing earlier. They didn’t look like that though.

[rapid hedge]

I think I know the issue with the cursor lock

It's probably my bad

[rapid hedge]

Worth noting that it happens on both Linux and Windows for me

Didn't test firefox yet

Doesn't crash on firefox

so Chromium-only

Yeah, I blame browser bug

let me know if you want to report it and need anything from me

[sturdy prawn]

Is that an em-dash I see? Clearly an AI comment (/s)

[slate scarab]

my phone autocorrects my double dash into it 🤖

[rapid hedge]

same lol

[rapid hedge]

Alright, Foxtrot 0.17 using the new audio backend and the patched thingy runs on itch!
https://janhohenheim.itch.io/foxtrot

All sounds seems in order now 🙂

Thanks for all the help and patience @slate scarab

[dusky mirage]

Web technologies Just Work™ 😎

Someone pointed out in the Firewheel discord server that it would be handy to have a "crossfade" node. That should be easy to do, so I'll work on that today.

[slate scarab]

ya that makes sense

[dusky mirage]

Oh yeah, btw did you want to comb through the Firewheel repository and add/remove bevy derives as needed?

[slate scarab]

I have not yet -- sorry for the delay. It's been quite busy lately for me! But I will make time for it here, at the very least before another full release.

[dusky mirage]

I also just realized there are a lot of nodes who's only config is to set the number of channels. It would probably make sense to just have a ChannelNodeConfig type.

That would also solve the confusion with DummyNodeConfig.

[slate scarab]

I do think there may be some utility in keeping different names, actually. The benefit being much better discoverability. But you can assess whether you value that aspect.

[dusky mirage]

Hmm, though I suppose that some nodes would want to have a different default number of channels.

[rapid hedge]

Oh I see y'all implemented HRTFs???

Wow!

I thought there was an issue with there not being a good Rust library?

[slate scarab]

ya cause you prompted me to go ask mr fyrox and he said sure

That's to say -- bevy_seedling exposes Fyrox's implementation. Fyrox's hrtf crate uses the IRCAM database, which is okay but rather small. Ideally, we'd be able to work with the SOFA format. That's Steam Audio's primary way to load HRTF subjects, for example.

However, SOFA is a bit of a monster to parse. @clear wasp started on it, but I think it would take a while to get over the finish line.

In any case, the current implementation should be a great starting point for anyone looking for more sophisticated spatialization. I had fun trying it out in foxtrot.

[rapid hedge]

oh whaaaaat

praise be onto Dima

[celest whale]

Nice! Good to see collaboration there

We should call that out in the release notes

[rapid hedge]

that's so cool 😄

[rapid hedge]

-# where PR?

[slate scarab]

i really do want to add a bunch of fun ambient audio / events to foxtrot
it would be a great testing ground for 3D stuff

[rapid hedge]

feel free to add as much as you want 😄

also tell me if you need any authoring tools

[slate scarab]

okay let me just commit 100mb of audio real quick

[rapid hedge]

like, place an object somewhere in trenchbroom and let it play sounds

[slate scarab]

right I remember discussing things like reverb zones

[rapid hedge]

I shall accept...

20

yeah, 20 MiB

[slate scarab]

haha okay that oughta be more than enough

[rapid hedge]

Yep 😄

there's also some generator models

those could emit sounds

[slate scarab]

generator models
what is generator model

oh

like

3d models of a generator

[rapid hedge]

yeah 😄

haha I see where the confusion comes from

[slate scarab]

listen it's late haha

[rapid hedge]

eh maybe more of a boiler

but the original Volta I map has some gadgets

sec, let me fire it up

they do funny beeps

these pipes here have hissing steam sounds

there's a field where crickets chirp

[slate scarab]

mm yes surely there's a crow in that field too

[rapid hedge]

CAW

Found it!

these guys do a neat whirring sound

(I still haven't figured a good way to record sound on Linux sorry)

[slate scarab]

okay i will imagine the whirring

[rapid hedge]

wait

I can dig it up

certainly an easy task

oh hey the crickets

[slate scarab]

yes good

the first one is so funny
why do crickets gotta slow down just a tiny bit every once in a while

[rapid hedge]

hehe

well I give up on finding the exact sound

but here's a good one

oh and this

[rapid hedge]

premium slightly ominous nondescript machine noise

oh, and this here plays a spark noise every now and then

lots of great CC-Non-commersial assets here 😄

Again, let me know if you need anything to experiment with 🙂

[slate scarab]

what are these assets btw?

the audio files

[rapid hedge]

https://www.thedarkmod.com/main/

All models, textures, and sounds I posted are from there

[slate scarab]

ohhh i see

[rapid hedge]

Foxtrot has also some CC0 assets from other places, but most are lifted from there

[limber kernel]

big hl1 vibes

[rapid hedge]

It's trying to keep the Thief 1 vibes, which shared a few devs and artists with HL 1 😄

good ear!

[limber kernel]

oh cool yeah that makes sense haha

[limber kernel]

random unimportant q: trying to take a stab at adding convolution since it seems straightforward (lmk if i shouldnt!) and was curious what you all would expect. BillyDM notes mix and enabled, which makes sense, but the impulse response also needs a spot. I was thinking something like this structure could work

pub struct ConvolutionNode<const CHANNELS: usize> {
    pub enabled: bool,
    pub mix: f32,
    pub response: Option<ArcGc<dyn SampleResource>>, // Stealing from sampler
}

I imagine the majority of use cases will be using the convolution node for applying a set IR for reverb effects. does it seem reasonable to use a SampleResource here too, as opposed to something like a raw buffer? (or something else entirely?)

[rancid elm]

if nodes are components then you can set per-node defaults with required components

[slate scarab]

The node configuration default mechanism is more general than its Bevy integration.

In other words, Firewheel doesn’t attempt to integrate with Bevy directly, and so this system should work well without Bevy at all.

[rancid elm]

ah

[pure dove]

just anecdotally, firewheel works quite nicely with macroquad too, makes it pretty easy to put together a simple interactive visualisation with sound without pretty much any boilerplate, ty for keeping it all engine-agnostic

[dusky mirage]

I get a lot of questions about how to stream audio over the network. I think we need a batteries-included solution, so I created an issue. https://github.com/BillyDM/Firewheel/issues/76

[slate scarab]

it would be so sick to have a robust network streaming crate in the ecosystem

[wary bridge]

The company I work for specializes in networked audio, so I may be able to get feedback on whatever solution

[steep dove]

don't we need a networking framework first?

[dusky mirage]

Maybe. Like I said I in the issue I don't know much about network programming.

[slate scarab]

i have to imagine that streaming audio is self-contained enough / has very particular requirements such that it wouldn't really fit in a general networking framework anyway

[dusky mirage]

Yeah, my guess is that we just need some API that requests/receives packets of opus data.

(And play/pause/seek)

[faint wigeon]

i have a bevy NDI implementation on working on but haven’t gotten to audio yet because was waiting for firewheel to land in upstream bevy. basically we just need a way to grab this frames slice of audio at the same time we submit the video

[faint wigeon]

at least for NDI all the networking is handled internal to the sdk

https://www.srtalliance.org/ is another newcomer open alternative i’m going to look more into as well

[steep dove]

sure you can put stuff in packets, but then do you need to dictate requirements about the transport as well. is out-of-order delivery acceptable? what about unreliable transport? many protocols batch together packets and periodically flush them, does a new audio audio packet need to trigger a flush?

[slate scarab]

I suppose my point there is that these questions seem to be completely different between audio/video and other networked data, to the point where abstracting over the two may not make sense. (Thus, the specialized libraries that Charlotte has shared.) But I also don't know what I'm talking about.

[wary bridge]

This is my general experience as well, with other networked audio software

[dusky mirage]

What is "NDI"?

[slate scarab]

https://ndi.video/ I believe

[faint wigeon]

oh yeah sorry 😅 it’s kinda the standard for doing networked av

for just audio over ip most things are handled via expensive interfaces so don’t need special handling

dante/avb

point being, at least in the commercial space, this isn’t really ever something you’d roll your own. another contender here in a different kind of direction is webrtc which i’m also hopefully going to build support for

@acoustic mulch may have more thoughts here too and has a lot of experience with pro av

[wary bridge]

We use Dante; mostly everything is handled for you, generally you just consume a buffer

[faint wigeon]

i use avb for lasers, but that likewise also just works with existing firewheel

[slate scarab]

generally you just consume a buffer
i like the sound of that

[celest whale]

omnomnom, buffers

[wary bridge]

No rust bindings afaik though, just c/c++ (for dante)

typical

[acoustic mulch]

Most of the stuff I've used is low latency, meant for just the local network, requires the network switches to have specific settings (Dante, AVB, AES67, etc...). Usually with these on a PC/Mac you'd have an audio driver that interfaces with it. So on the bevy side you'd want support for things like ASIO. On hardware usually the manufacture buys an interface card with a chip (eg. from audinate) and then use something like I²S. (Though there are exceptions)

[faint wigeon]

yeah there’s asio support already, although now i’m remembering i may have made to make some small patches to get avb working, ill need to check

[acoustic mulch]

Nice! Having asio would be sick. The audio latency in OSU kills me as a musician. It's like 10x longer than I'm used to vs using asio. You can obviously be good at OSU without low latency, but it would be so much more satisfying. And I'd love someday to work on a rhythm game.

[acoustic mulch]

What is this API for? Software dante, or something that interfaces with DVS or VIA directly?

[wary bridge]

It's a part of Dante Embedded Platform

[acoustic mulch]

Oh interesting! Hadn't seen this yet. Looks like software dante lite-ish (lower channel count, higher latency) but setup to more easily license

[wary bridge]

Yea that's basically it

[dusky mirage]

Fascinating. I wonder how they implemented this? https://www.youtube.com/watch?v=WALpap5ZyuI

https://github.com/WhoStoleMyCoffee/raytraced-audio

Ah ok, it looks like they are doing the ray tracing on the main thread, and then using the result to set the parameters of a reverb, lowpass filter, and volume/pan.

So something like this can already be done in Firewheel!

Or wait, actually we need a reverb node first.

[dusky mirage]

I wonder what reverb algorithm Godot uses?

(My guess is the FreeVerb algorithm)

[dusky mirage]

Adding a freeverb node to Firewheel should be pretty easy, since we can mostly copy Fyrox's implementation. https://github.com/FyroxEngine/Fyrox/blob/master/fyrox-sound/src/effects/reverb.rs

[slate scarab]

i have one in seedling actually

((copied from somewhere else))

https://docs.rs/bevy_seedling/latest/src/bevy_seedling/nodes/freeverb/mod.rs.html

[dusky mirage]

Oh, sweet!

[slate scarab]

if you're chill with the implementation, we can just move it over
idk if we already have all passes

and combs

so we might be able to strip those out

[dusky mirage]

I think it makes sense to just have all the DSP part of the reverb node. Different reverb algorithms may have different implementations of those filters anyway, so stripping them out probably isn't worth it.

[slate scarab]

they're quite simple anyway

[dusky mirage]

Cool. Though I noticed that the mix parameter isn't smoothed, and that it doesn't make use of silence optimizations. I can help with that.

[slate scarab]

im not sure there's much to be done for silence

How can you reliably determine whether the reverb has converged to silence without excessive overhead?

[dusky mirage]

Yeah, we'll just have to manually check to see if the output contains silence if it didn't contain silence in the previous processing block. It will be worth it though, especially if you have a pool of worker node chains with reverbs that don't do anything most of the time.

[slate scarab]

I suppose there are two main approaches

1. Heuristics -- cheap, but easy to mess up
2. Analysis -- less cheap, but necessarily correct

[dusky mirage]

Manually checking for silence isn't actually that expensive, considering that you can break out of the loop as soon as you detect a non-zero sample.

[slate scarab]

Presumably you could at least skip the analysis when the input isn't silent.

yeah i suppose one scan through the buffer isn't that bad

even compared to a cheap reverb

[dusky mirage]

Correct. The only time we need to perform the check is if the input silence flags are set and the previous block was not silent.

The silence check will look like this (I should probably add this to firewheel-core)

pub fn is_silent(buffer: &[f32], amp_epsilon: f32) -> bool {
    let mut silent = true;
    for s in buffer {
        if s.abs() > amp_epsilon {
            silent = false;
            break;
        }
    }
    silent
}

Oh yeah, there should also be a parameter to pause, resume, and reset the reverb.

(Otherwise when you pause the game, the reverb will still ring out.)

Although maybe that's not that big of a deal.

[slate scarab]

no that would definitely be bad haha

luckily that seems pretty trivial though

[dusky mirage]

Hmm, and you would also need to add declicking for pause/resume.

There is certainly a lot of boilerplate for silence optimizations, parameter smoothing, and declicking. I wonder if there is some kind of wrapper we can make to make it easier? 🤔

I'm not sure about parameter smoothing, but I think it should be possible to create an easy to use wrapper for silence optimizations and declicking.

[dusky mirage]

Hmm, never mind. I'm not sure there is a good generic way to do that.

Ok, I've added the method ProcBuffers::check_for_silence_on_outputs which should be helpful for this.

[slate scarab]

did you want to move over the freeverb node before the next release?

[dusky mirage]

Sure. A reverb node would be very useful.

And so all you need to do is "If the input silent flags are set and the previous block output silence, then skip processing. Otherwise, process the reverb. If all input silent flags are set and ProcBuffers::check_for_silence_on_outputs returns ProcBuffers::ClearAllOutputs, then mark the previous block as being silent.

Oh actually, I've needed to keep track of the previous block being silent for other nodes. It would make sense to have the Firewheel engine automatically keep track of this for you in ProcInfo.

[dusky mirage]

Ok, I've added a ProcInfo::prev_output_was_silent field

[dusky mirage]

Oh yeah, serde derives. I should do that too.

[faint wigeon]

do we have a plan for upstreaming this wonderful work during 0.18? i recall there being some talk about a demo / head to head with the current implementation

apologies if i missed anything (:

[slate scarab]

i do have a work-in-progress little demo that compares Firewheel vs rodio directly, and alice mentioned she could help see it through in the next cycle

here it is for reference https://github.com/CorvusPrudens/rust-audio-demo

it's a cute mini narrative with just a few interactions

I'd say the only real thing missing is maybe a bit of a stress test.

I've done these informally on my own (and found Firewheel heavily favorable over both kira and rodio), but I haven't baked in anything properly.

[limber kernel]

oh nice, so i imagine the same would be helpful in the convolution node for determining if the buffer is silent?

[dusky mirage]

Yeah, that's what I was referring to.

[limber kernel]

speaking of convolution - wanted to share a quick demo. I think the clicking noise on sample changes is due to the method provided for loading a new IR sample, which clears the internal state of each convolution buffer, but will investigate further. Also the wet signal is very loud haha, but it's hard to tell if its just constructive interference with the sine/white noise input.

(warning: loud)

really want to try hooking this up to foxtrot and mixing between different IRs to simulate reverb

[slate scarab]

oh you have an ir node?

[limber kernel]

just the convolution part, im loading in IR samples.

honestly unsure how it would work in bevy. maybe different impulses can be defined by points or volumes, and interpolated between. ill take a search since im sure theres some talks or blog posts about it

[drifting dawn]

Is it intentionally that this soundtrack fading example here (https://bevy.org/examples/audio/soundtrack/) is not using bevy::audio::Source::fade_out() at all?

[vocal dock]

Some games do this lol

[limber kernel]

Did you take a stab at play/pause in reverb yet? I am going to do the same for the convolution node & wondering how you think it's best to expose (maybe using the SharedPlaybackState in sampler? or just a bool?)

[slate scarab]

a simple boolean as a part of the parameters seems ideal

then you'd probably just do a tiny bit of declicking and basically stop processing the input stream

presumably, the input to the reverb will have a bit of declicking of its own (if it's being fed by samplers, for example), so it should all match up

[limber kernel]

yeah i think that makes sense. if someone wants to completely stop and remove any internal state, that's probably a case to just re-intialize the node.

[slate scarab]

oh you could have a Notify<()> field for that

this would be a lot more efficient than recreating it tbf

[limber kernel]

ah, i think that's out of my current understanding. need to do some more reading of the repo

[slate scarab]

Notify is a special wrapper type that will always produce a patch when it's accessed, even if the internal value doesn't change. This is useful for a number of things, such as a playhead parameter.

In this case, it serves as a convenient way to "trigger" an event that doesn't have a payload. You'd do something like this:

#[derive(Diff, Patch, Default)]
struct ReverbNode {
    // other parameters...
    pub pause: bool,
    pub reset: Notify<()>,
}

// To reset it, you'd call `Notify::notify`
let mut node = ReverbNode::default();
node.reset.notify();

Then in the processor you can handle the ReverbNodePatch::Reset(_) variant.

[limber kernel]

ah i see, makes sense. appreciate the explanation

seems kinda like bangs in puredata

[slate scarab]

ya

very analogous

[rapid hedge]

Alright friendos

I did everything in https://github.com/janhohenheim/thief_sense_demo that I could before absolutely needing to tackle audio

This means I'll tackle the bevy_seedling + audionimbus story this weekend

@cold isle did you already start working on something in that regard?

I don't have enough time on my hands to do a full Bevy abstraction

But my goal is to setup a new repo containing a minimal example of how to integrate audionimbus with bevy_seedling

First issue

I fail to run the audionimbus demo

that's with cargo run -p demo --features audionimbus/auto-install

If I instead download Steam Audio 4.6.1 and place the right libphonon.so in /usr/local/lib (and $LD_LIBRARY_PATH for good measure) and then run cargo run -p demo, I instead get this:

which apparently means that the audio context failed?

can anyone here help me out with how to run the audionimbus demo?

Oh I see, the docs.rs description is out of date, you actually need 4.7.0 (Submitted a PR )

[rapid hedge]

alright, now I get this funky cpal error on cargo run -p demo too 👍

Apparently this line is using different buffer lengths for output and staging_buffer

where the buffer is 4 times the size of the output

Hmmm weirdly enough https://github.com/MaxenceMaire/audionimbus-demo/tree/master runs fine

maybe I should start by updating that to Bevy 0.17 instead But I'm afraid of just running into the exact same crash once I update audionimbus too

[slate scarab]

oh boy haha

does cpal actually require interleaved audio?

[oak walrus]

Last I checked, the answer is "unspecified" because it forwards the platform's buffers and the layout of cpal stream's buffers is undocumented, but it seems to be yes, it's interleaved

[cold isle]

did you already start working on something in that regard?
Sorry I don't have the full context, did you mean integrating with Bevy?
@past sandal developed an FMOD integration and has a PR open to migrate to AudioNimbus (https://github.com/GitGhillie/bevy_fmod_phonon/pull/6). I tested it and it worked great.

Someone showed interest in a Bevy<>AudioNimbus plugin (https://github.com/MaxenceMaire/audionimbus/issues/1), and I'm more than willing to help but I'm missing a lot of context on the current status of the audio part of Bevy. I've seen Firewheel mentioned, but if I remember well the plan was to build a native spatial audio solution instead.
And yes, cpal is a mess to integrate with, especially because it takes buffers of varying sizes even if you request a specific size 😅 I'm working on a game prototype myself, so I came up with a middle layer to queue Steam Audio buffers (and an audio scheduler with multiple buses that can be paused/resumed). I'm happy to share it if there's interest, although I suspect Firewheel already does something similar

Also I need to udpate the AN demo, it still uses the old version 🙂 do let me know if you need any help integrating it in the meantime

[slate scarab]

if I remember well the plan was to build a native spatial audio solution instead

This would be ideal purely from a toolchain perspective (non-Rust deps can be annoying to build or block platforms like the web). However, this is a prohibitive amount of work for the moment. While upstreaming something built on audionimbus directly might not happen, a robust third-party crate integrating it with Bevy would be a great addition to the ecosystem!

Oh and to be clear, I'm talking specifically about a sophisticated spatial audio solution like Steam Audio. We have pretty basic stuff like simple HRTFs in the works.

[cold isle]

I agree, FFI linking can be tedious, better to keep that away from Bevy 🙂
And by all means, let me know how I can best help. I do have a lot of catching up to do, though

Also @past sandal was working on a complete Steam Audio rewrite, not sure what the progress is but it looked promising

[slate scarab]

oh yeah that's phonon, right?

[cold isle]

Yep, that's the one: https://github.com/GitGhillie/phonon_rs

[rapid hedge]

Specifically bevy_seedling <-> audionimbus, yes

bevy_seedling is what's planned to become Bevy's new audio backend in case you're out of the loop

[slate scarab]

(which itself is built on Firewheel)

[rapid hedge]

my most immediate question is: how do I fix the main audionimbus demo lol

[rapid hedge]

I'll try to upgrade the Bevy demo on my own now

Pretty sure I'll fail, but I'll let you know the errors hehe

[rapid hedge]

Huzzah!

I ported the Bevy demo and it works 🎉

[cold isle]

None that I'm aware of, no. But we can have a crack at it 🙂

[rapid hedge]

https://github.com/janhohenheim/audionimbus-demo/tree/bevy-0.17

[cold isle]

Awesome!

[rapid hedge]

I took the liberty of turning the character controller into a simple flycam to cut down on boilerplate

want a PR?

[cold isle]

Yes, please! I'll take a look at soon as I get back home, thanks

[rapid hedge]

There ya go: https://github.com/MaxenceMaire/audionimbus-demo/pull/1

Okay, now the harder part: How the heck does this work with seedling

@slate scarab I remember you said something about that

But I can't remember

[slate scarab]

well it's still on the rc so that doesn't help 😅

[rapid hedge]

eh that's fine

semver compat

[slate scarab]

(i'll be migrating this weekend)

[slate scarab]

wait is it

[rapid hedge]

yep

Foxtrot is on 0.17 right now, depending on bevy_seedling with no issues

[slate scarab]

oh nice

[rapid hedge]

@slate scarab mind taking a little look at this file here https://github.com/janhohenheim/audionimbus-demo/blob/bevy-0.17/src/audio.rs and giving me a few pointers on where to start?

I can't act on them right now because I'm going to a birthday party and need to prepare a presentation for PowerPoint Karaoke 👀

But I'm excited to do the seedling integration when I'm back

[slate scarab]

okay cool i'll need a couple hours before i can get to it

[rapid hedge]

oh, that's good timing then 😄

[slate scarab]

Whether it can truly be integrated depends on the shape of the steam audio API, and how that meshes with independent audio processing nodes. I haven't looked into the API at all, so I'm just crossing my fingers haha. I guess we'll find out soon though.

[rapid hedge]

the file I sent should be fairly representative I hope

[slate scarab]

okay i think there's a very nice blend of steam audio and firewheel that we can tease out here

ambisonic encodings will be no problem -- in this demo, the ambisonic encoding breaks out the audio stream into nine channels, but we can just route all nine however we want within firewheel

i.e. we could sum them all up and decode them in some terminal ambisonic decoder buffer

[slate scarab]

Ah, yes apparently it's vastly more efficient to only decode once, which we could easily do.

You're only running the HRTF once that way.

ambisonic orders or electron orbitals?

[steep dove]

spherical harmonics

[slate scarab]

waves are waves ig

[steep dove]

it's the eigenfunctions of the laplacian again

one of the coolest parts of modern mathematics

[slate scarab]

Okay so I think we'd want an overall graph like this (lmk if it doesn't look right, I'll just use an image if so):

┌──────────────────────────┐         
│SamplerNode               │         
└┬───────────┬────────────┬┘         
┌▽─────────┐┌▽──────────┐┌▽─────────┐
│ReverbNode││ReflectNode││DirectNode│
└┬─────────┘└┬──────────┘└┬─────────┘
┌▽───────────▽────────────▽┐         
│VolumeNode                │         
└┬─────────────────────────┘         
┌▽───────┐                           
│PoolBus │                           
└┬───────┘                           
┌▽───────────┐                       
│AmbisonicBus│                       
└┬───────────┘                       
┌▽──────────┐                        
│MainBus    │                        
└───────────┘                        

The SamplerNode will be a normal Firewheel sampler, probably outputting just a single channel. The reverb, reflect, and direct node all represent Steam Audio processors that turn this mono signal into an ambisonic stream. Let's say it has nine channels. Those nine channels can be mixed together and routed to a volume node so the sampler's volume can be individually adjusted. This sampler will be part of an ambisonic pool, so it'll route into the pool's volume node (PoolBus). Then, we'd have a single decoder in the graph that all ambisonic streams are routed to. This would decode those nine channels back down to two (or whatever the setup is). Then, the simple stereo stream would get routed out to the output like normal.

It would be very easy to separate out the fancy spatialized sounds from the normal, non-spatialized ones (like UI elements or non-diegetic music). If you want fancy spatialization, just play your sound in the fancy pool.

There is one thing here that we can't really do at the moment, though -- in the diagram, the stream splits into three and then recombines. Currently, you can only express direct chains in bevy_seedling for per-sampler effects, so we'd probably just combine these three into a single node that does the job of all three. That's probably more efficient anyway.

I believe all the spatial simulation could be done in the ECS (and should be done this way when occlusion is involved anyway), and then the results could be picked up with Firewheel's diffing and sent to the steam audio nodes.

All that to say -- this seems like a natural fit for Firewheel's node-based processing model. It should just slot right in!

[rapid hedge]

Heck yeah!!

I also need to think how to get the information out as data instead of as sound to play

For the AI system

[slate scarab]

i think that might depend on how much information you can get from steam audio's apis

The simulation data should all be in the ECS, so it's not locked into the audio thread or anything. But idk if steam audio gives you reflection / occlusion data that's easy to work with

[limber kernel]

lmao i found this topic the other day when researching true stereo mode. it seems very cool but way out my understanding

[limber kernel]

you could be saying made up words and i would not know 🧠

[celest whale]

It's always the eigenfunction of the laplacian 😩

[cold isle]

Steam Audio can also do non-diegetic if it makes it any easier https://docs.rs/audionimbus/latest/audionimbus/effect/binaural/struct.BinauralEffectParams.html#structfield.spatial_blend
https://valvesoftware.github.io/steam-audio/doc/capi/guide.html#spatial-blend

You can use Steam Audio to blend between spatialized and unspatialized audio. For example, a radio playing in the distance can be spatialized (or diegetic), but as the listener moves closer, the sound can become less spatialized, until eventually it becomes part of the soundtrack (or non-diegetic).

I'd have to look at the C code but I doubt setting the spatial blend to 0.0 skips the spatialization compute altogether, so I suppose it'd be better to separate them as you mentioned

I was wondering, can Firewheel create buses/channels (not sure what you call them) at runtime, for example when changing the audio output from stereo to 5+ channels in-game?

[slate scarab]

Yeah, definitely.

"Buses" in Firewheel are more of a concept rather than an actual API. It's a node graph, like Max or Pure Data if you've ever used those. You can route arbitrary outputs to arbitrary inputs.

[cold isle]

That's great! And does it solve the problem of varying buffer sizes cpal has? Or is it up to the user to choose and adapt the audio backend? That's long been a pain point for me, especially when integrating Steam Audio

If yes, that'd be a great fit 🙂

[slate scarab]

Sorry, at the risk of asking you to repeat yourself -- what's the issue you're running into here?

How does a varying buffer size cause issues with steam audio? Does it require a fixed size?

[cold isle]

Yes, Steam Audio expects a fixed number of samples (if I remember well it has to do with tail handling). And even when requesting a specific size (say 1024), cpal sometimes provides a different size

[slate scarab]

Yeah this is platform dependent -- sometimes the system just gives you different buffer lengths. I don't think it's so much cpal as the system itself.

I just wanted to clarify because I've run into similar issues myself, specifically with FFT processors that want a fixed buffer.

[cold isle]

May I ask how you solved it? In my case I just went with a custom scheduler that keeps a queue of samples and feeds them into the cpal buffer to match the exact size

Before doing that it typically resulted in audio "cracks"

[slate scarab]

Firewheel itself doesn't solve this issue (indeed, it can be exacerbated by finely-scheduled events). But it's not especially hard to solve. There are two ways to guarantee fixed buffer sizes:

1. Write a little custom backend for Firewheel that uses buffering to guarantee fixed buffer sizes at the graph level. As long as your fixed buffer chunks aren't significantly larger than the actual buffer size, you won't get crazy latency or anything. (Indeed, in the common case where the block size is the same as what you requested, you won't get any additional latency.) You could basically just copy/paste the default cpal backend and add some buffering to achieve this.

However, this comes with the caveat that you'd have to disable fine-grained scheduling for the whole graph to guarantee that the block size doesn't change. This isn't super ideal since this fine scheduling is quite powerful for nodes that can accept it.

2. Manage this buffering at the individual node level. Basically the same as above, but managed within each atomic processing unit in the graph. This is less efficient, but it should be pretty negligible overall. This has the benefit of not placing any constraints on event scheduling for the rest of the graph.

I did (2) for the HRTF nodes I integrated.

We could probably write a helper for this guaranteed block sizing.

[cold isle]

Interesting, thank you. I suppose keeping e.g. one frame buffered (meaning having a latency of 1024 samples) would not be noticeable and it would easily solve the problem indeed (unless cpal's buffer is systematically larger than Steam Audio's length). I'll have to play around with Firewheel to understand your second point and how the node architecture works exactly, but it sounds interesting. I'll try and find time this week-end 🙂

[rapid hedge]

@slate scarab before anything fancy, how would I just replace the part in the demo that uses rodio with seedling?

The stuff that depends on use rodio::{OutputStream, Sink, Source};

Or maybe with firewheel directly

[slate scarab]

well the demo is actually just writing to the sink in the ECS

that would be like making a Firewheel node that has a ring buffer, and just writing to that from the ECS

[rapid hedge]

Is there an easy example for that?

[slate scarab]

no 😅

[rapid hedge]

I'm very much a total beginner in audio haha

[slate scarab]

hmmmm

okay actually i think it would be pretty simple

the easiest thing to do would be to basically dump almost everything in that big method into a single Firewheel node

it should have one input and two outputs

maybe two inputs is easier

[rapid hedge]

One input is the sample I presume

[slate scarab]

dang, okay maybe it's not that easy 😅 sorry it's late here

uh

[rapid hedge]

Hehe no worries

Worst case I can also use rodio until someone spoon feeds me code haha

[slate scarab]

Really you'd want to define two nodes: a spatial processor, which essentially is that entire function (minus the last bit) and then a decoder (which is the last bit).

if i had just a little more time i coulda outlined it more clearly
but alas 😩

Just for some context, this part: https://github.com/janhohenheim/audionimbus-demo/blob/221d9114c7c62ef6ff0a568aa8f0f863800408c9/src/audio.rs#L202-L237 gets the raw sample data. You don't need that -- this is just given to you as input to your node in Firewheel.

[rapid hedge]

Do you have some time tomorrow? No pressure!

[slate scarab]

This part https://github.com/janhohenheim/audionimbus-demo/blob/221d9114c7c62ef6ff0a568aa8f0f863800408c9/src/audio.rs#L239-L272 calculates all the parameters for the steam audio Source type. This should be done in the ECS. The results should then be sent to the audio processor.

It would be pretty easy to write a Diff / Patch wrapper that just sends the whole damn thing when anything changes haha.

[rapid hedge]

Alright, I get the rough outline

But very little clue about how to actually write that

I'll peruse the firewheel rxamples 🙂

[slate scarab]

This is where it actually gets interesting. https://github.com/janhohenheim/audionimbus-demo/blob/221d9114c7c62ef6ff0a568aa8f0f863800408c9/src/audio.rs#L291-L310 This turns the mono audio stream into the nine-channel ambisonic stream. If you wanted to start small, you could probably calculate just this inside a Firewheel node.

So you'd have a node with one input and nine outputs haha. And then you'd have a corresponding decoder with nine inputs and two outputs (the decoding happens here https://github.com/janhohenheim/audionimbus-demo/blob/221d9114c7c62ef6ff0a568aa8f0f863800408c9/src/audio.rs#L383-L393).

But ya i'll be able to help more tomorrow for sure!

[rapid hedge]

thanks a bunch, really appreciate it!

[rapid hedge]

what are the two outputs in this case?

[slate scarab]

left and right 😎

[rapid hedge]

oh

that makes sense lol

This might correlate to my two ears then

[rapid hedge]

Apparently I need to depend on firewheel_core directly to use Patch and Diff:

Is that intentional?

[limber kernel]

are you on firewheel 0.8.0-rc.1 or main? I dont seem to see that on the released version

[rapid hedge]

neither, bevy_seedling = "0.6.0-rc.1"

well, now I am on firewheel-core = "0.8.0-rc.1" to fix it

I just kinda thought that enough would be re-exported by seedling

but I suppose not

[rapid hedge]

Thanks for the hint though, looks like I can drop the _core 🙂

[limber kernel]

oh i see what you're getting at, yeah it might make sense to re-export those in seedling

though if youre making custom nodes you might need to be bringing it in anyways

i suppose assuming seedling is upstreamed, ideally it would be something that users can interact with without needing to think about firewheel, so a solid base library of nodes to cover most use cases would probably mitigate the need to start making your own/bringing in firewheel as a dep

[rapid hedge]

What's seedling / firewheel's way of sharing things across nodes? In my case, things like the Context and some others only need to be created once, but used by all nodes

Do I just use Arc<RwLock<T>>?

Or what about the audionimbus::Source? How does my node get that?

Does it store it on its own? If so, I can't meaningfully derive Patch / Diff for that, can I?

[rapid hedge]

Ooooh or do you mean that the whole audio source business should happen in a regular Bevy system and then set the relevant fields on the node?

that makes sense!

Then the audionimbus source can just stay a component, right?

[rapid hedge]

But this here again needs an annoying type, namely that pesky AmbisonicsEncodeEffect

I suppose I need to store that in the node?

If so, is there a way to tell the patch / diff derives to just ignore that field? Given that there's isn't really anything to patch, right?

and also, the processor should store all buffers as fields, right? I ask because I read somewhere that construct_processor is safe to allocate in, which implies to me that I shouldn't allocate in process

[limber kernel]

not sure on the other qs, but yes def dont allocate in process as it can cause crackling from underruns

[rapid hedge]

@cold isle am I right in my understand that if I want to know roughly from where a sound came for AI purposes, I need to do a pathing simulation?

If so, am I correct in that I need to manually place the probes around the level for that to work?

Or is there some way to get that "where did the sound come from" from the ambisonics sound field?

@slate scarab another question for you, how do I get information that was calculated in the processor back to the ECS?

[slate scarab]

This would require the firewheel macro crate to be aware of bevy_seedling, which is dubious. It’s a bit annoying, but yeah you need to depend on Firewheel to implement nodes.

I mean technically you could just bring firewheel_core in scope but

[slate scarab]

Easiest way would be if you could share it with atomics (and use the shared state that you can configure with the audio node info.

See the LUFs node for reference in seedling

otherwise you’ll need some primitive that ideally doesn’t allocate, like a fixed cap channel or something

[slate scarab]

No, it should only exist in the audio processor. It’s the effect params that should exist in the node. Furthermore, the setting struct should exist in the config. This is because it contains the max_order field, which likely sets how many buffers to allocate on construction.

[drowsy dome]

use trotcast

you can do what corvus says ofc, but trotcast or crossbeam would work

(i would like it if u used trotcast but yknow, there's only one real user of the channel rn, ur lookin at em)

okay actually for this usecase use crossbeam

trotcast holds a mutex lock on senders

but if you wanna broadcast that info, I can make it so it doesn't hold a lock

I just got lazy

in theory though the lock should not be held for more than a few nanoseconds

[slate scarab]

that'll crash on Wasm regardless of how long it is

[drowsy dome]

how so?

well actually

im about to find out

[slate scarab]

aborts

[drowsy dome]

who says its gonna panic

[slate scarab]

well it doesn't panic haha

[drowsy dome]

there shouldn't be any deadlock

[slate scarab]

Any atomic.wait on the main browser thread or in an audio worklet will immediately and violently abort execution. It's the runtime that does it, not Rust. So there's nothing you can do.

[drowsy dome]

that's so hype

well

I have a SOLUTION!

no_std uses a spinlock hehehe

[slate scarab]

ya a spinlock would work

[drowsy dome]

again, the spinlock should also only spin and acquire for a few nanoseconds

but I gotta make it a feature

but it will totally block you if you use .send over .try_send and the channel is full. I think that's good behavior. nonetheless, I would like to eventually convince corvus with enough birdseed that maybe trotcast has a place in seedling (once I get rid of the mutex ofc)

[oak walrus]

The problem isn't how long the lock is held, but how long the OS is going to respond to your locking request; and the answer to that is, relatively, a long time, because the OS might decide to pause your thread and do something else for a while. That also happens when you try_lock

[drowsy dome]

are you serious? well that's fucked. I did not think it would park

without a condvar

[oak walrus]

Any syscall is a park opportunity

[drowsy dome]

...mutexes require a syscall

[slate scarab]

ya which is why they tricky for audio

[drowsy dome]

well shit I feel like a fucking clown, that's ridiculous

ok til

I'm definitely going to remove it then

[slate scarab]

these days it's, ah.... less problematic certainly (rodio uses them everywhere haha) but it's not best practice

[drowsy dome]

good is the enemy of perfect

[slate scarab]

true true

that's how the saying goes

[drowsy dome]

the saying has it backwards, good enough requires enshittification acceptance in practice

please don't tell me atomic exchanges require syscalls

[oak walrus]

rodio using mutexes everywhere is 95% the reason why we performance of Bevy Audio is so poor

[slate scarab]

I actually rather doubt that's the case. Uncontended mutexes should be very fast. And they're mostly uncontended.

But maybe all the locking adds up.

[oak walrus]

That and wasm is single-threaded which means the audio processing has to share compute with the rest of Bevy

[slate scarab]

I’d like to note that it’s not like rodio’s performance is horrendous. In my testing, I found it to be 2-3 times slower than Firewheel. But kira is only a little bit faster than rodio.

[oak walrus]

I don't know, on my end I have never been able to make rodio work on the web without glitches on my MBP M4, whereas seedling has never as much as glitched once

[slate scarab]

oh yeah well it’s definitely not good on the web because of the single threaded situation

that’s definitely horrendous 😅

And I think the mutexes basically deny the possibility of multithreading on the web.

[cold isle]

By "know from where a sound came", do you mean inferring the direction of the source based on the Steam Audio output? Couldn't you simply use the source position for that? Or is it because you want to avoid the situation where the source is behind a wall and the direction would point toward the wall instead of following the sound waves bouncing off walls?
Usually SA is used to process sound, I've never actually seen it used the other way around to read back direction metadata. You could technically achieve this using ambisonics though, since it encodes the sound field directionally

I need to do a pathing simulation?

Pathing combines multiple paths from the source to the listener, as opposed to simply taking the shortest straight line between the two, so I'm not sure if that's what you're directly after?

[rapid hedge]

Oh ok, I thought maybe the pathing output included the probes it used

[cold isle]

Hmm I don't think there's a way to get that information, no. Also probes are used for baking, but you can still apply the effects without them

[rapid hedge]

Yeah the AI system I'm implementing originally used a custom built propagation system that allowed for asking where the most recent sound bounce came from

[cold isle]

Ha interesting, and yeah I can definitely see the use for it. I think your best bet would be ambisonics in this case, although I'll have to check if there's a way to decode information from it

[rapid hedge]

Thx

[rapid hedge]

(It’s the AI from Thief in case you’re curious 🙂 )

[cold isle]

I'm definitely super curious 🙂 Thief as in the Thief game? I've never had a chance to play it, but I'm tempted to give it a try now

[rapid hedge]

Yep, that one. I looooove that game and dug deep to find out how exactly that AI works 😄

Can highly, highly recommend

[rapid hedge]

Having some trouble finding that

[rapid hedge]

How do I get that into the processor in practice? Creating that type requires access to the audionimbus context.

Is there a way to share the context globally with all nodes?

[cold isle]

Love it! All the more reason to find a solution 😉

I checked the docs but to the best of my knowledge SA doesn't provide an easy way to directly decode the source direction. My intuition would be to work with first-order ambisonics (with higher order ambisonics, we probably want to project it to a first-order one to make things easier) so that we get the four channels w (pressure), x (left-right directional component), y (front-back), z (up-down). X/Y/Z normalized would give us an estimation of the direction the sound is coming from. Also I would keep an average over a short window of frames since I suppose it would be too noisy otherwise. That's just my two cents though, I'm not sure how sensible that approach would be

[rapid hedge]

From my rudimentary understanding, that sounds about right 🙂

But in a pinch I could also just use the real origin, the most important but is how much the sound is muffled by materials

e.g. when a player walks on carpet in a hallway they’re really silent, and when they’re walking on a metal grate in a large hall their sound travers quite far

And for that the final intensity that steam audio provides is already enough

[slate scarab]

sorry, renamed to loudness

[slate scarab]

It’s just the internal state of the processor— you’d construct it in the AudioNode trait.

[rapid hedge]

But then that node needs the context

How does it get it?

Do I store it in the ECS, wrapped in some kind of shared spinlock?

[cold isle]

Yup, and you can easily swap the listener's position with the AI's to get the intensity from their position

[slate scarab]

Well, everything just takes a shared reference to it, so… it could just be a oncelock.

A mutex/rwlock would also be fine. It won’t ever be passed to the audio thread anyway.

AudioNode::construct is never called in the audio thread.

[rapid hedge]

Aaah gotcha

[rapid hedge]

But wait, then I need to store that locked context in the node, right?

So it can build the processor with it

Can I exclude that field from the diff / patch derives?

[slate scarab]

No, as a once lock it could just be static.

[rapid hedge]

Ooooooh I see

Okay that makes sense

[slate scarab]

(But yes, you can ignore fields in the derive)

[rapid hedge]

So it’s not in the ECS at all

[rapid hedge]

Cool! do you happen to know the syntax?

I couldn’t find it at a glance

[slate scarab]

maybe i should update the docs 😅

[rapid hedge]

I can dig deeper if you don’t happen to know it off the top of your head 🙂

[slate scarab]

#[diff(skip)]

If you don’t like the once lock, you could also store it in the node’s config. Although that’s a bit suspect at the moment because I require Default. I should probably adjust that.

There’s probably something to be learned here for figuring out the best way to integrate something like Steam Audio. Some way of passing state around that doesn’t result in awkward situations.

[rapid hedge]

@slate scarab

• is it intentional that usize is not Diff / Patch?
• It says struct AudionimbusEqualizer<const N: usize>(pub [f32; N]); is not RealtimeClone. Imo this shouldn't allocate on clone (fixed size array), so it's safe to just add that derive too, right?
• Should something containing a Mutex be RealtimeClone? Or, like, used in a processor at all? @dusky mirage said there should not be any mutexes at all due to parking, right? Then how do I turn something that is not Sync into Sync then? Some spinlock crate?
• I see seedling likes to use ArcGc. Should I use that too over Arc?

@cold isle how come you don't wrap the SimulationOutputs in a dedicated plain old Rust non-FFI struct? I'm running into the issue that the SimulationOutputs as-is are non-Send, so I need to wrap them manually in a copy-pasted struct containing the direct params etc.

[rapid hedge]

hmm, looks like I have to manually wrap a lot of types to give them Diff / Patch. Curse you, Orphan Rule!

Okay bit of a roadblock I cannot fully wrap ReflectionEffectParams because it contains audionimbus_sys::IPLReflectionEffectIR, which is non-Send.
@cold isle any tips? Can this just be made Send maybe? How about Sync?

Note: this needs to be Send in order for me to construct it in the ECS and then send it over to seedling / firewheel

[rapid hedge]

Next question for Max: can I fill a previously created AudioBuffer with new data? I need to fully preallocate it.

Preallocating the data is trivially ofc, but I see that calls to AudioBuffer::try_new also allocate here:

And finally for today, question for everyone: how do I turn an AudioBuffer or its inner Vec<f32> into a channel-split &[&[f32]]? I assume I can just split it into 9 (= number of channels) chunks of equal size? And I probably don't need to call either AudioBuffer::interleave or AudioBuffer::deinterleave on that since I need the "raw" data to continue anyways, right?

Current progress: https://github.com/janhohenheim/audionimbus-demo/blob/seedling/src/audio.rs#L35-L155

[slate scarab]

if those types truly aren’t Send, that’ll be really annoying!

If they’re !Send just because they contain a pointer, that’ll be an easy fix

[slate scarab]

1. Partly, since it's variably sized. I suppose we could just stuff it into the u64 variant for all practical purposes.
2. Yeah, it would be reasonable to do so
3. nahhhhh, although the solution depends on why you need Sync. We had an Exclusive-equivalent wrapper at some point, although it may have been removed
4. ArcGc makes it easy to move things around without accidentally dropping them in an audio processor. If you have some Arc resource that you send to an audio processor, that's a good candidate.

[slate scarab]

There's definitely some waste going on here from the perspective of Firewheel. AudioBuffer's safe API seems to be expecting a flat buffer with each channel arranged one after the other. It then allocates pointers to each channel.

This is undesirable within Firewheel because 1. it allocates, and 2. it needlessly allocates! The input and output buffers of a Firewheel processor are already arranged as Steam Audio expects.

(Please correct me if I'm wrong!)

It seems like you can't currently sidestep this either, since the unsafe constructor takes a Vec.

[rapid hedge]

1: Why is it a concern that it’s variable sized?
3: Simply because the diff / patch / component macros yell at me when the pointer I mentioned above is not Sync

To expand on this, I'm like 90% sure that pointer is should to be Send, but I have no clue if it’s Sync

[rapid hedge]

Mind expanding on this?

The part about firewheel arranging the buffers already correctly

Does that mean we can just add a fairly trivial constructor to audionimbus?

[slate scarab]

1. Because we want to maintain careful control over the size of the intermediate representation enum. Although it has pointers in it already, so we might be able to add a usize field without increasing its size. But then again, do we add isize as well?
   \3. You could wrap it in that Exclusive-equivalent to make it Sync, although yeah Send is not so trivial.

[slate scarab]

Actually, it's not exactly how steam audio wants it. In a Firewheel processor, you have &[&[f32]] as inputs and &mut [&mut [f32]] as outputs. In other words, a slices of slices.

It looks like steam audio wants a pointer to float pointers. If the wide pointers of the slices start with the pointer (and not the length), then I suppose you could very unsafely cast them to pointers to float pointers.

But either way, I think the AudioBuffer should take a slice of these pointers, not an owned Vec. The data it refers to is borrowed anyway, so the owned Vec doesn't really provide any benefit except for mild convenience.

Conversely, it forces you to allocate while processing the audio, which isn't ideal.

[slate scarab]

Oh btw a small note on the snippet you shared

You shouldn't need to wrap AudioSettings and put it in the node or anything -- you can construct it either in AudioNode::construct_processor or in the process method itself if required. The sample rate and frame size are given to you in both locations.

Not that important, but it's at least one less thing to deal with!

[rapid hedge]

Do you happen to know the name of the Exclusive-like?

[slate scarab]

@dusky mirage removed it in favor of OwnedGc.

which would maybe work here?

[rapid hedge]

As long as it turns Send into Send + Sync I'm happy 🙂

[rapid hedge]

I think I should be able to do something cursed

namely transmute (&[T], usize) into a Vec, then std::mem::forget the buffer so it doesn't double free

[slate scarab]

that is cursed lamao

[rapid hedge]

hehe

[slate scarab]

i mean it'll still result in a free though won't it

[rapid hedge]

why?

[slate scarab]

oh you mean you'll forget the whole buffer

[rapid hedge]

bingo

[slate scarab]

AudioBuffer

[rapid hedge]

exactly

[slate scarab]

[rapid hedge]

yes.

[slate scarab]

ya i think that would work

although that's gotta be way worse than just tanking the allocation haha 🤣 it's not great but it'll be very unlikely to cause problems on its own

[rapid hedge]

yeah no, this should definitely be an actual method upstream haha

[slate scarab]

do it though
for science

don't even consult miri

[rapid hedge]

I never consult miri

-# Never bothered to set it up in CI lol

[slate scarab]

i've actually managed to run miri on an entire (end-to-end) test, complete with bevy_ecs and a mock audio backend

it took a couple minutes but it did do it

[rapid hedge]

that's impressive

pub trait CursedAudioBuffer<T> {
    fn cursed_new(channel_ptrs: &[*const f32], num_samples: usize) -> audionimbus::AudioBuffer<T>;
    fn cursed_new_mut(channel_ptrs: &[*mut f32], num_samples: usize)
        -> audionimbus::AudioBuffer<T>;
}

impl<T> CursedAudioBuffer<T> for audionimbus::AudioBuffer<T> {
    fn cursed_new(channel_ptrs: &[*const f32], num_samples: usize) -> audionimbus::AudioBuffer<T> {
        let i_will_not_mutate_pinky_promise = unsafe { std::mem::transmute(channel_ptrs) };
        Self::cursed_new_mut(i_will_not_mutate_pinky_promise, num_samples)
    }

    fn cursed_new_mut(
        channel_ptrs: &[*mut f32],
        num_samples: usize,
    ) -> audionimbus::AudioBuffer<T> {
        let capacity = channel_ptrs.len();
        let pseudo_vec = (channel_ptrs, num_samples);
        let cursed_vec: Vec<*mut f32> = unsafe { std::mem::transmute(pseudo_vec) };

        unsafe { audionimbus::AudioBuffer::try_new(cursed_vec, num_samples) }
    }
}

// SAFETY: this is UB lol

[slate scarab]

good good yes

[dusky mirage]

• Oh yeah, I forgot about usize/isize. I'll add that real quick.
• Yeah, it should be safe to derive RealtimeClone for that.
• Mutexes are a big no-no in realtime code. Why do you need a mutex?
• ArcGc is preferable since it ensures that resources won't be dropped on the realtime thread.

[rapid hedge]

3. is because otherwise the derive macros yell at me because a pointer is not declared to be Sync

[dusky mirage]

Ok, what are you using a pointer for then?

[rapid hedge]

Check the convo above

[slate scarab]

steam audio ffi

[rapid hedge]

I would recap for you, but I'm about to shower haha

[dusky mirage]

Oh ok. Well if you know that the pointer won't be accessed concurrently, then you would need to use unsafe impl Sync {}.

[slate scarab]

yeah, but that would be upstream in audionimbus

[rapid hedge]

Can also do it downstream in a wrapper

[dusky mirage]

Then I guess you would need a wrapper.

[slate scarab]

Well, actually we couldn't just assume that unless the compiler enforced it

Without writing easily unsoundable code anyway

Anyway, we'll have to look into the thread safety of steam audio's types

[rapid hedge]

So if I have a pointer in a processor, is it guaranteed by firewheel that it won’t be accessed concurrently?

FWIW I used OwnedGC for now

[dusky mirage]

Well, you have to make that gaurantee if you are doing FFI bindings.

Or are you asking if node processors themselves are Sync? Node processors are only Send, not Sync.

[rapid hedge]

Lol I have no clue what the FFI bindings exactly are doing haha

[dusky mirage]

Hmm yeah. Doing something like this will definitely take a lot of work.

[rapid hedge]

But I pinged Maxence earlier, they will know 🙂

[rapid hedge]

I would hope to be able to do a minimal integration for now. Now fancy interop library, just a way to hack seedling / firewheel together with audionimbus so that it plays sound 😄

I believe I'm on a good path rn with Corvus' help

[slate scarab]

Yeah a simple integration should not be especially extensive. At a high level, all we're doing is placing a handful of steam audio processors in Firewheel nodes.

[dusky mirage]

Oh, be careful when doing this. A pointer to slices is different than a pointer to floats.

This is how you convert a slice of slices to a pointer to pointers:

let inputs: ArrayVec<*const f32; MAX_CHANNELS> = buffers.inputs.iter().map(|b| b.as_ptr()).collect();
let ptr = inputs.as_ptr();

[rapid hedge]

Oh btw Billy I absolutely love your profile pic haha

[slate scarab]

It depends on the layout of the wide pointer.

If the pointer in the wide pointer is first, then it should "appear" as a pointer to pointers of floats.

I don't think the layout is guaranteed though.

Which is annoying.

[dusky mirage]

Oh ok, you might be right. Still, it's probably better to be on the safe side.

[slate scarab]

ya

[rapid hedge]

This doesn’t allocate, right?

[dusky mirage]

Correct. ArrayVec doesn't allocate.

ArrayVec is a very useful crate for realtime code.

[rapid hedge]

Thanks, then that’s exactly the snippet I needed

[slate scarab]

It doesn't, but you can also do this with a vec just fine if you pre-allocate the number of channels you need when the node is constructed.

If the node happens to need a flexible number.

[dusky mirage]

Oh yeah, that too. ArrayVec just happens to be easier when constructing a slice of slices due to lifetimes.

But raw pointers don't have lifetimes, so you can use a preallocated Vec.

[slate scarab]

Ah yeah because you'd really have to do some work to convince the borrow checker you're not mutably aliasing any of the buffers when you try to borrow all of them mutably.

[dusky mirage]

Well the main problem is you can't store the preallocated Vec in a struct.

[slate scarab]

mm well that's also a problem if it's a Vec<&mut T> yeah 😅

there's probably a fancy type we could come up with that handles this

[dusky mirage]

There is techincally a hack to transform a Vec<&'static T> into a Vec<&T> and vice-versa, but ArrayVec is just easier.

(You have to do some mem::swap and unreachable!() shenanigans.)

Oh wait, mem::swap isn't needed. Still, the hack that doesn't need unsafe looks like this:

struct Foo {
    bar: Option<Vec<&'static [T]>>,
}

impl Foo {
    fn process(&mut self) {
        let mut bar = self.bar.take().unwrap();
        bar.clear();
        let mut buffers: Vec<&[T]> = bar.into_iter().map(|_| unreachable!()).collect();
        
        {
            /// ... use buffers
        }

        buffers.clear();
        let mut bar: Vec<&'static [T]> = buffers.into_iter().map(|_| unreachable!()).collect();
        self.bar = Some(bar);
    }
}

Hmm, though actually I suppose you could create a wrapper type that returns a gaurd.

[slate scarab]

yeah that's what i was thinking

[dusky mirage]

Ah yeah, I think this could work.

pub struct PreallocVec<T: 'static> {
    buffer: Vec<&'static mut T>,
}

impl<T: 'static> PreallocVec<T> {
    pub fn new(capacity: usize) -> Self {
        Self {
            buffer: Vec::with_capacity(capacity),
        }
    }

    pub fn borrow<'a>(&'a mut self) -> PreallocVecRef<'a, T> {
        let mut tmp: Vec<&'static mut T> = Vec::new();
        std::mem::swap(&mut self.buffer, &mut tmp);

        PreallocVecRef {
            data: tmp.into_iter().map(|_| unreachable!()).collect(),
            buffer: &mut self.buffer,
        }
    }

    pub fn borrow_mut<'a>(&'a mut self) -> PreallocVecMut<'a, T> {
        let mut tmp: Vec<&'static mut T> = Vec::new();
        std::mem::swap(&mut self.buffer, &mut tmp);

        PreallocVecMut {
            data: tmp.into_iter().map(|_| unreachable!()).collect(),
            buffer: &mut self.buffer,
        }
    }
}

pub struct PreallocVecRef<'a, T: 'static> {
    pub data: Vec<&'a T>,
    buffer: &'a mut Vec<&'static mut T>,
}

impl<'a, T: 'static> Drop for PreallocVecRef<'a, T> {
    fn drop(&mut self) {
        let mut tmp: Vec<&'a T> = Vec::new();
        std::mem::swap(&mut self.data, &mut tmp);
        *self.buffer = tmp.into_iter().map(|_| unreachable!()).collect();
    }
}

pub struct PreallocVecMut<'a, T: 'static> {
    pub data: Vec<&'a mut T>,
    buffer: &'a mut Vec<&'static mut T>,
}

impl<'a, T: 'static> Drop for PreallocVecMut<'a, T> {
    fn drop(&mut self) {
        let mut tmp: Vec<&'a mut T> = Vec::new();
        std::mem::swap(&mut self.data, &mut tmp);
        *self.buffer = tmp.into_iter().map(|_| unreachable!()).collect();
    }
}

I might go ahead and create and publish a small crate for this. 😁

[limber kernel]

basically a vec that cant be allocated after creation?

[slate scarab]

Yes, with the added constraint that it preallocates capacity for items that would otherwise have troublesome lifetimes.

[limber kernel]

i was thinking something like that would be very helpful for specifying channels over config

since i need to use a vec to store some number of channels, but i dont want it to be resized for obvious reasons afterward

but that might be a diff problem

[slate scarab]

oh i actually have a simple fixed capacity vec in seedling

well, fixed after construction

[limber kernel]

cool cool

[slate scarab]

https://docs.rs/bevy_seedling/0.5.5/bevy_seedling/utils/fixed_vec/struct.FixedVec.html

but it's so simple you can just kinda paste it wherever

[rapid hedge]

hmm transmuting random shit into a Vec is not as easy as I thought

still need an "allocator"

that obviously doesn't do anything

guess I'll just fork audionimbus for now

[slate scarab]

honestly i kinda can't believe people are okay in 2025 with not really knowing how thread safe their types are
i don't want to have to dig through the api docs to find out if maybe someone took the time to clearly communicate whether something is Send and or Sync

Anyway, I'm pretty sure everything is probably Send, but a number of things definitely aren't Sync.

[rapid hedge]

same

uuh

how2impl

[slate scarab]

oh that must be in the audionimbus-sys crate

[rapid hedge]

I think it needs a wrapper

[slate scarab]

oh that's just an alias

[rapid hedge]

yep

[slate scarab]

yeah that should probably be a newtype

[rapid hedge]

oh I see, the current pattern in the crate is to mark the types that use the pointers as Send

So it's just unsafe impl Send for ReflectionEffectParams {}

fair enough

@slate scarab you agree that this should be a *const *mut Sample instead of a Vec?

though that would be probably a bit annoying

[slate scarab]

I only have a limited view of the crate so far, but yes, I don't think there's much utility in the Vec. The API should probably have the user pass in a slice of *mut Sample.

[rapid hedge]

ah so not even a pointer to pointer?

[slate scarab]

Or the raw *const *mut Sample.

Either one -- the former would be a mild convenience.

[rapid hedge]

@cold isle ping in case you happen to be online 🙂

[slate scarab]

While it would be annoying, it's tough to avoid. The current API forces allocation, which imo isn't worth it for a bit of convenience.

[rapid hedge]

And how should I convert the slice of ptrs to a ptr?

[slate scarab]

well that you can actually just cast

you can coerce a slice into a mere pointer

it's just a slice of slices that's tricky

[rapid hedge]

oh wait the buffer has methods to read the length of the stored stuff

so it may want to actually store a slice itself

but that would introduce lifetimes everywhere

[slate scarab]

well at some point it's converted to the steam audio representation, which wants a pointer and a length

[rapid hedge]

hrmmm

[slate scarab]

But those lifetimes are probably good anyway. In practice it'll already have a lifetime.

Since you're passing in something that implements AsRef<[Sample]>... like &[Sample]!

[rapid hedge]

Not AsRef<[*mut Sample]>?

ah well I can cast one to the other I spose

[slate scarab]

(i mean for this method https://docs.rs/audionimbus/latest/audionimbus/audio_buffer/struct.AudioBuffer.html#method.try_with_data_and_settings)

[rapid hedge]

what should I do here?

thanks for your patience btw 🙂

[slate scarab]

oh no worries haha, the alternative would be that we don't get anything going with steam audio for a while so im down!

yeah i mean so in this case you're kinda toast

The arrayvec approach would work for some maximum set of channels

But I'm not sure this method is worth preserving.

[rapid hedge]

There's also the hacky alternative

just add a constructor that allows you to pass that ugly vec

ah no wait that still wouldn't work

because it would take it by ownership

[rapid hedge]

I'll just yeet it and see what the compiler says

welllll it's called by try_with_data_and_settings and try_with_data

But I suppose I could change those to take in a &[*mut Sample]

[slate scarab]

yaaaaa i kinda think so

[rapid hedge]

hmmm now I need to fix the tests that currently take Vec<f32> to use &[*mut f32]

this is ugly

oof

I need an adult

[slate scarab]

tbh i didn't think it would be an extensive change 😅

[rapid hedge]

It's entirely possible I hecked up

could you check my diff?

sec

https://github.com/MaxenceMaire/audionimbus/pull/20

[slate scarab]

to be honest it looks good to me

[rapid hedge]

that's nice

[slate scarab]

is try_new even unsafe now?

i don't think it is

[rapid hedge]

not according to the compiler, but it wasn't before either

[rapid hedge]

hrmm

[slate scarab]

What's the problem here though?

[rapid hedge]

how do we make a neat API for this now

[slate scarab]

hold on i've almost got a PR up for Firewheel

[rapid hedge]

All usages look like this

The usage currently expects a &[f32], which is nice

way nicer than &[*mut f32]

wonder if we can have a constructor that takes in &[&mut f32]?

[dusky mirage]

Hmm, ran some tests and it only works when compiled in release mode. On debug mode it still allocates.

[slate scarab]

what if you do reserve_exact in the constructor? idk if that helps

[dusky mirage]

That's not the problem. The problem is that the tmp.into_iter().map(|_| unreachable!()).collect() trick appears to only work in release mode. In debug mode it allocates a new Vec.

[slate scarab]

i'd think a little unsafe should do the trick there

[dusky mirage]

Oh wait nevermind. It's working now.

(It was panicking at something else, derp)

[rapid hedge]

fuck it

time to just pass in a &mut [*mut f32] as a buffer for internal usage

[slate scarab]

Yeah we could probably make a little utility to do this

[dusky mirage]

Ok, here's the implementation I came up with! I haven't added any docs yet. https://codeberg.org/BillyDM/prealloc_ref_vec/src/branch/main/src/lib.rs

[slate scarab]

Here's the Freeverb PR with all the bells and whistles https://github.com/BillyDM/Firewheel/pull/78

[dusky mirage]

Cool! I'll take a look at it here soon. (I suppose I should get off my computer and eat lunch with my family.)

[rapid hedge]

@slate scarab wheew that took a while

check this https://github.com/MaxenceMaire/audionimbus/pull/20/files

[slate scarab]

ya that seems like a decent compromise! hopefully @cold isle agrees haha

[cold isle]

Lots of messages to catch up on, let me go through them real quick 🙂

how come you don't wrap the SimulationOutputs in a dedicated plain old Rust non-FFI struct?

Hmm, it’s been 8 months since I wrote it, but it looks like I did it to enforce lifetimes - although looking at it now, there don’t appear to be any pointers. I suppose we could copy its contents into a pure Rust struct and deallocate it right away.

Can this just be made Send maybe? How about Sync?

We probably can if we carefully handle lifetimes - I don't see an easy way to make IPLReflectionEffectIR into a rusty type directly since SA manages its data internally without exposing it. But we should be able to build something around it to make ReflectionEffectParams sendable

can I fill a previously created AudioBuffer with new data?

Yes totally, AudioBuffer is a bit of a misnommer since it's simply a pointer to the sample data

how do I turn an AudioBuffer or its inner Vec<f32> into a channel-split &[&[f32]]?

Maybe something like this?

(0..num_channels)
    .map(|channel| {
        let start = channel * num_samples_per_channel;
        let end = start + num_samples_per_channel;
        &v[start..end]
    })

Maybe this should be a helper by the way

And I'm totally cool with changing the Vec of pointers if it makes it any easier for your integration - as you mentioned it's purely for convenience, to avoid having to specify the lifetime everywhere. But it does force an extra allocation, which is a bummer if it's done repeatedly as it seems to be the case with Firewheel

[rapid hedge]

check the draft PR 🙂

@cold isle is it intentional that the reverb effect and the reflection effect seem to be the exact same thing in the demo?

[cold isle]

Taking a look now. I'll probably need to come back to it tomorrow with fresh eyes though 🙂

[rapid hedge]

yeah no rush!

[cold isle]

Definitely not haha. This is the work of someone tired 😅

[rapid hedge]

hehe

I noticed there is no audionimbus::ReverbEffect

so how does one do reverb?

[cold isle]

It looks great on first read! I think we could make the API a bit cleaner by moving the prepare_channel_ptrs logic into a separate constructor. But overall, this seems like the right direction. I'll put my mind to it tomorrow or early this week 🙂

[slate scarab]

i was wondering that haha

[cold isle]

Is the code you shared part of the demo inside the AN repo? I think the actual demo contains an example, let me check

[rapid hedge]

It's from https://github.com/MaxenceMaire/audionimbus-demo

[rapid hedge]

yeah sounds good

[cold isle]

Hmm actually looking at the code it looks correct - reverb also uses the reflection effect, you just need to position the listener at the source position

[rapid hedge]

where do you do that positioning?

also, here's the seedling WIP: https://github.com/MaxenceMaire/audionimbus-demo/pull/2

[cold isle]

It's the parts using listener_source.source e.g. here https://github.com/MaxenceMaire/audionimbus-demo/blob/master/src/audio.rs#L188-L189
This one is onnly for reverb, the rest uses audio_source

Steam Audio briefly explains how to get reverb in their doc here https://valvesoftware.github.io/steam-audio/doc/capi/guide.html#reverb

[rapid hedge]

Ooooh I see, thanks!

@slate scarab If I use 3 nodes, each returning 9 outputs, how do string those together?

Do I have a node with 27 inputs?

[slate scarab]

No you can sum all three streams of each channel on another node with 9 inputs.

[rapid hedge]

oh lovely!

[slate scarab]

The ambisonic encoding can be summed and multiplied like a normal audio stream.

You will have to specify the connections manually right now (we only automatically handle this for stereo connections in bevy_seedling right now, but we'll be able to infer the correct connections by default soon).

[rapid hedge]

do you have an example for this

[rapid hedge]

and for this?

🙂

[slate scarab]

omg like the one method i don't have an example on

https://docs.rs/bevy_seedling/latest/bevy_seedling/edge/trait.Connect.html#tymethod.connect_with

you'll give it a hefty &[(0, 0), (1, 1), (2, 2), ..., (8, 8)]

The tuple represents (output index, input index)

[rapid hedge]

and that adds them together?

For context, I'm thinking about how to translate this into firewheel:

Next question

Is it intentional that the glam types like Quat don't implement Patch / Diff?

huh looks like Vec3 does implement it

But Transform / Quat do not

I suppose that's an accident then?

[slate scarab]

No, we specifically only implemented it on a small subset of types.

[dusky mirage]

Ok, published the crate! https://crates.io/crates/prealloc_ref_vec

[rapid hedge]

So I have to create my own little SeedlingTransform wrapper?

Should the firewheel outputs be interleaved or deinterleaved?

[slate scarab]

de

[rapid hedge]

I'm close to something compiling, I think 👀

[slate scarab]

i was in a call with bird

okay what's happening

[slate scarab]

Yeah I mean we can definitely expand the number of types that we derive for. Vec2 and Vec3 were of immediate concern because we use them (or related types) directly.

[rapid hedge]

Also, this^ 🙂

[slate scarab]

Oh right hold on

[slate scarab]

The connections are describing how you connect one node to another. Like

SamplerNode -> Volume
            ^^ connection

If you connect multiple sampler nodes to one volume node

SamplerNode --> Volume
SamplerNode ^

They're automatically summed together at the input of the volume node.

[rapid hedge]

ah, cool!

[slate scarab]

Since nodes can have multiple inputs and outputs, you may need to describe exactly which channel goes where. Like you might want to "cross" the left and right channels of a stereo node.

[rapid hedge]

So to implement the per-effect gain, I would first connect each node to a volume node (or something like that) and then sum them?

[slate scarab]

Yes that would allow you to control each individually

[rapid hedge]

Okay, my bs compiles

now let's set up the nodes and pools

[slate scarab]

Yeah so I hope I mentioned this earlier -- while it would be ideal if we could separate these effects out into three nodes.... it's actually kind of annoying at the moment 😅

[rapid hedge]

hehe I see

I did one one for the moment

[slate scarab]

Because you can only describe a serial chain with sample_effects![]

[rapid hedge]

and then one for decoding

[slate scarab]

oh well that's fine

[rapid hedge]

Mind checking out this file now? https://github.com/janhohenheim/audionimbus-demo/blob/f2ef8e245fe9660b055efca85b46859c4bf0a699/src/audio.rs

I realize that SAMPLING_RATE and FRAME_SIZE should be encoded in firewheel

[slate scarab]

ya they're given to you where you need access to them

[rapid hedge]

but other than that, I hope I did it somewhat correctly

I'm honestly a little bit confused which parts I should best create in the ECS and then pass to the node and which parts I should create in the node and pass to the processor

I did a weird mix rn

For example, right now the node creates the AmbisonicProcessor::ambisonics_encode_effect and co from its own audionimbus context

but I could instead build them in the ecs, then pass them to the node, which passes them to the processor

[slate scarab]

i think this makes sense since they only need to exist in the audio processor

The idea with AudioNode is you have a lightweight, trivially cloneable (mostly) type that doesn't have any special "links" to its processor.

This makes it very easy to use in the ECS -- you don't have to be careful with where any particular AudioNode came from.

All you need to associate it with the audio processor in the Firewheel context is its NodeID.

aw man i wish i could do a pr review on it, i have a couple notes so far

[rapid hedge]

why can't you?

[slate scarab]

it just a repo

[rapid hedge]

oh sec

[rapid hedge]

https://github.com/MaxenceMaire/audionimbus-demo/pull/2

[slate scarab]

pog

[rapid hedge]

@slate scarab to tell seedling to spawn the ambisonic node and connect it to the decoder, do I need to do sample_effects![(A, B)] or sample_effects![A, B]?

[slate scarab]

latter -- it's just a wrapper around a related macro basically

each node in the ECS should be a separate entity

(And you can add other components like the node's configuration when spawning them this way.)

Although.... damn, actually I don't think it'll work currently.

The pools expect the effects chain to have a consistent channel count that matches the sampler output.

[rapid hedge]

welp that's not the case here

[slate scarab]

I could fix this right now

Before the next (hopefully imminent) release

[slate scarab]

That is, when node connections aren't given a specific channel configuration, we can do three things:

1. If the number of output and input channels between the nodes match, just do 1-1 for the whole deal
2. If they don't match, just connect whatever number overlaps both.

3. i ran out of things

Automatically upmixing/downmixing is actually not the best idea by default in my opinion.

[rapid hedge]

@slate scarab I hooked up the nodes

The PR is "done"

as in, it doesn't work but it compiles 😄

I need some heavy help from this point on :>

[slate scarab]

i know you made an issue about this im sorry please forgive me for my sins

[rapid hedge]

I did?

[slate scarab]

You need to register your nodes haha

[rapid hedge]

That rings a bell

[slate scarab]

https://github.com/CorvusPrudens/bevy_seedling/issues/44

oh i guess i made it
on your behalf 😅

[rapid hedge]

OOOOOH

I remember 😄

Now I get
2025-10-05T22:43:28.604042Z ERROR bevy_seedling::edge::connect: failed to connect audio node to target: Input port idx 1 is out of range on node NodeID(Index { slot: 42, generation: Generation(1) }) with ChannelCount(1) input ports

which is what you mentioned just now, right?

[slate scarab]

Yeah by default it attempts stereo, instead of checking the I/O. And while you could fix this particular error, it would then enforce one channel for all of the effects (which absolutely won't work 😅).

[rapid hedge]

How do I get the frame size when constructing the processor?

(also what's a frame size lol)

[slate scarab]

the size of the audio buffers

a "frame" is one "row" of the audio stream -- in other words, if we have a stereo audio stream, each frame is composed of a left and right sample

[rapid hedge]

gotcha

[slate scarab]

At a sample rate of 44.1kHz, you have 44.1k frames regardless of how many channels there are. So "buffer size" isn't totally accurate, since it may actually be bigger when there are more channels.

[rapid hedge]

while you're on it, what exactly is a channel? Is that like a "layer" in images?

[slate scarab]

Sound is how we perceive pressure waves in the air. When we measure sound, we usually have some microphone that records the pressure experience by a single transducer in space. To encode it digitally, we measure this pressure 44 thousand times or so a second. This single microphone produces a single "channel" of audio -- just a stream of pressure readings.

But we have two microphones on our head -- two "channels." So we often manipulate audio with two channels.

[rapid hedge]

Makes sense! I'm just wondering why steam audio has 9

[slate scarab]

oh well that's cause they get freaky with it

[rapid hedge]

hehe

[rapid hedge]

I suppose StreamInfo has the frame size somewhere?

[slate scarab]

it has a max size, but this is actually a bit tricky

i'll explain in a sec

[slate scarab]

i don't understand details to be honest, but those nine channels allow them to encode the spatial information of a sound in a kind of intermediate representation

They basically represent "pressure fields" in 3d space. This allows you to sum a ton of these encoded sounds and then decode them into a stereo signal (using an HRTF) in one go.

The alternative would be applying the HRTF eagerly to every sound, and then summing the outputs of the HRTFs.

But this is much less efficient.

[rapid hedge]

Got it

Thanks

(I'm learning sooo much about audio hehe)

[rapid hedge]

the reason I'm asking is...

...to remove these self.settings here

which contain just the sampling rate and the frame size

[slate scarab]

Okay so you can get the maximum number of frames https://docs.rs/firewheel/latest/firewheel/struct.StreamInfo.html#structfield.max_block_frames, but Firewheel does not guarantee a consistent number of frames. On the contrary, Firewheel often provides nodes deliberately smaller buffers so that it can give them sample-accurate events via ProcEvents.

This is a problem, because Steam Audio needs perfectly consistent frames. Luckily, (or unluckily??) this would be a problem regardless. @cold isle was talking about it earlier -- cpal won't always give you the number you ask for, and sometimes it'll seemingly randomly change it. This is because some hardware / OSes switch it up from time to time.

The best way to fix this is to buffer the input and output of the steam audio processors (at the cost of potentially a bit of latency and extra memory) so they're always consistent.

I did this for Fyrox's HRTF, because it also needs consistent sizes.

You can see that here where I fill the input buffer and here where I drain the output.

You basically just add a couple vectors, construct them with whatever fixed size you want to use, and then you're off.

[rapid hedge]

So you push stuff every frame(?) until it hits fft_buffer_len, and then process that?

[slate scarab]

Yes (don't worry it's not icky -- you can reserve the correct capacity so it doesn't allocate).