Better Audio

[rapid hedge]

so does that mean I would construct the ambisonics_encode_effect during the processing?

[slate scarab]

No that would all still remain as-is. You can just choose some size like 256 and only ever use that.

Or better yet -- put the size in a config struct so people can tune it if they want.

256 is a decent default because the frame count is unlikely to be smaller (and if it is it probably won't be latency sensitive anyway).

[rapid hedge]

OOOOOH and then I let the processor only process once it has reached 256 frames?

Okay I think I got it

alright, I'll work on removing the mentions of AudioSettings which hardcode the sample rate and frame size

And if I understood you correctly, there's currently nothing I can do to get it to run, right?

Because of the output size

[slate scarab]

well you totally could, you just can't slot it into a sampler pool yet

if you add the effects after the pool, you can freely connect them however you like.

[rapid hedge]

how do I do that 👀

or am I better off just waiting a bit until there's an API upstream?

[slate scarab]

commands
    .spawn(SamplerPool(Steamy))
    .chain_node(AmbisonicProcessor::default())
    .chain_node_with(AmbisonicDecoderNode::default(), &[(0, 0), (1, 1), ..., (8, 8)]);

commands.spawn((
    Steamy,
    SamplePlayer::new(server.load("wow.wav")),
));

This will apply the same exact effect to all sample players in Steamy, but ya know it would at least work.

I should have enough time in the morning to add the proper feature for this to work within the sampler pool.

[rapid hedge]

thanks

Hmm

I need the Simulator in the ECS

But to construct it, I need the sample rate and frame size

which I just put into an AmbisonicsNodeConfig

which would be nice

but now I need that exact information earlier

[slate scarab]

Do you need the simulator before constructing these nodes?

[rapid hedge]

hmmmm

I suppose not, no

But I need one simulator globally

not one per node

so uuh

do I just pick the config of the first node?

[slate scarab]

Right, my point is that they can be decoupled with no problem. If the frame sizes need to match, then I'd just hardcode it for the library as a const. To get the sample rate, you can just ask what it is from the audio context when constructing the simulator.

[rapid hedge]

how do I get the sample rate in the ECS?

[slate scarab]

two ways, hold on

You can just ask it via the resource:

fn system(mut context: ResMut<AudioContext>) {
    let sample_rate = context.with(|c| c.stream_info().unwrap().sample_rate());
    // ...
}

But this is annoying (the stream might not be active, you won't catch restarts, etc.).

The better way would be to observe the StreamStartEvent https://docs.rs/bevy_seedling/latest/bevy_seedling/context/struct.StreamStartEvent.html and StreamRestartEvent https://docs.rs/bevy_seedling/latest/bevy_seedling/context/struct.StreamRestartEvent.html.

They'll tell you exactly what the sample rate is the moment it initializes (or changes).

The sample rate can change when you change devices.

Like if you unplug headphones or whatever.

[rapid hedge]

Ooooh I see

so I should rebuild my simulator each time I change the sample rate, right?

[slate scarab]

yeah if you can't change the sample rate with a method

[rapid hedge]

good point, let me check

nope, no chance

but yeah rebuilding sounds fine

[rapid hedge]

oof

when creating a simulator, all sources need to be added

so rebuilding it in practice means migrating it

@cold isle is there some method you could add so change a simulator's sampling rate at runtime?

[rapid hedge]

little issue with this: there are no sane defaults for the processors

I mean, I could just wrap all params in Options

or hmm

I guess I can rearrange

[slate scarab]

well in that example you could provide actual values

i was just using some shorthand

[rapid hedge]

But to generate those, I need the sample rate

Which I don't have on startup

(assuming you meant to put that snippet on startup)

[slate scarab]

nothings stopping you from spawning it in an observer

[rapid hedge]

fair enough

[slate scarab]

(StreamStartEvent will only trigger once, so that should be a good thing to listen to.)

Also it’s basically always triggered in PostStartup, so it’s basically still at startup.

[rapid hedge]

ah neat

eek

please tell me I don't need to implement Component for the processor

I assume you meant to use the nodes there and not the processors, right?

[rapid hedge]

No dice:
2025-10-06T00:58:51.032601Z ERROR bevy_seedling::edge::connect: failed to connect audio node to target: Input port idx 1 is out of range on node NodeID(Index { slot: 42, generation: Generation(1) }) with ChannelCount(1) input ports

[rapid hedge]

oooh is this because I'm actually now reading an ogg which is stereo?

[rapid hedge]

Doing this doesn't insert AmbisonicProcessor into the entity holding the pool, so I'm unsure how to retrieve them. The query I wrote doesn't match as a return :/

Anyhoot

I addressed all your feedback 🙂

See the updated PR

The one thing I didn't do is react to changed sample rates

Because that would be very very annoying, and I hope we can get a function to just set that instead

[rapid hedge]

The demo runs, but because the query never matches any nodes, nothing plays.

Oh hey I just figured I can hack it by setting the settings on all nodes

hmm now audionimbus tells me that the input must have exactly one channel

I set it to two because I thought maybe seedling is expecting me to accept two channels from the ogg

Alright, changed it back to use a single input channel and used this thingy to make seedling accept my setup:

    commands
        .spawn(SamplerPool(AudionimbusPool))
        .chain_node_with(ambisonic_node, &[(0, 0), (1, 0)])
        .chain_node_with(
            ambisonic_decode_node,
            &[
                (0, 0),
                (1, 1),
                (2, 2),
                (3, 3),
                (4, 4),
                (5, 5),
                (6, 6),
                (7, 7),
                (8, 8),
            ],
        );

And now it successfully sounds like a buzzing fly 😄

huzzah!

given that it sounds like a weird fly no matter how far away the camera is, I think it's safe to say I fucked something up

pushed that as well

Enough for today

If you want to try it on your machine, you'll need to also clone https://github.com/janhohenheim/audionimbus/tree/no-alloc, including the submodule

[rapid hedge]

never mind, couldn't resist playing around some more 😄

It now plays sound!

It's clearly still a bit borked

but it's recognizable

Interestingly it sounds more or less correct when the camera flies out of the room Though it never becomes more silent with distance

[rapid hedge]

Oh, another thing:

if proc_info.in_silence_mask.all_channels_silent(inputs.len()) {
    // All inputs are silent.
    return ProcessStatus::ClearAllOutputs;
}

would this not kill remaining reverb after an input ends?

[slate scarab]

oh did i not actually send that feedback

[slate scarab]

yes

it would

[slate scarab]

yes sorry

[slate scarab]

Yeah, it shouldn’t — the pool already has a node when you spawn it (a volume node). When you chain a node, it spawns a new one and connects the first entity (in this case the pool) to it.

You can supply arbitrary bundles to chain_node, so you could give it a marker if you want.

[rapid hedge]

Not on GitHub, but you wrote plenty on Discord for me to work with

Can i get the connected nodes' entities?

[rapid hedge]

^

because I assume that every sample player that uses that pool would spawn their own downstream nodes, right?

So I would need to have a way to go from the pool entity to the downstream node entities in order to query them

I saw that using effects would give me a relationship for just this

[rapid hedge]

Or is there a misconception here, and this syntax would use the same node entity for all things with the pool?

[slate scarab]

Yeah, the snippet I suggested just creates one for the moment. All sample players from the pool would feed into that one effect. So not a particularly useful setup, but again it works at the moment 😅

[rapid hedge]

Aaaah then everything makes sense, thanks

(This could use some documentation :p)

[slate scarab]

i think it probably is mentioned in the docs

[rapid hedge]

Oh, then my bad!

[slate scarab]

or maybe it's implicit (to me anyway haha)

The pools do a bit of abstraction and "magic," which might not help with the mental model for normal node connections.

[slate scarab]

You can right now, but it's a bit annoying. Really, we'll need many-to-many relationships to do this properly.

[rapid hedge]

When I first used seedling I thought that a node pool was like a thread pool

Hehe

But it’s more like a node collection

[slate scarab]

(You have to get the edges from the audio context and compare them to the FirewheelNode on the entities, etc.)

[rapid hedge]

or at least, in my mental model, a node is like a commit, a pool is like a branch, and a bus is like a tag

In git terms

[slate scarab]

that's quite the analogy 🤣 can't say i've thought about it in git terms

[rapid hedge]

My mind works in mysterious ways

[slate scarab]

it do kinda be like a thread pool though

[rapid hedge]

orly?

[slate scarab]

You "queue" tasks (SamplePlayers) which have to wait for availability. When there's a slot, they claim it (the actual SamplerNode), and then free the slot when they're done.

The queue is pretty short though; the SamplePlayer will have the PlaybackCompletionEvent fired after just a hundred milliseconds or so if it doesn't find a slot.

(So you don't build up a long queue of sounds that exhibit crazy latency -- they're instead just skipped.)

[rapid hedge]

Oh, I thought every sample player would have their own lil node

That they then use

[slate scarab]

In a sense! But it's acquired from the pool. If it spawned a new node every time, the graph would have to be recompiled for every sound.

Thus, why we have pools.

[rapid hedge]

Oooh I see

[slate scarab]

I do have it set up to allow the pools to grow though if they run out of capacity. They grow quadratically.

That's why their size is described as a range.

(initial capacity, max capacity)

[rapid hedge]

I was thinking how I should use the nodes later in practice to describe the audio output per npc

Since the take in the listener location as a param

Do I just pass in a vector of listeners instead, fill some output field on the node according to that list, then only use the player's location for the actual node output?

In practice, the NPC calculations don’t need to be in the privileged audio thread

They don’t care about audio stutters I believe

[slate scarab]

Hmmmmmmmm sorry I'm not quite sure what you mean

describe the audio output per npc

[rapid hedge]

Each NPC needs to know the amplitude of the sounds that are incoming at their location for the stealth system

The node setup right now only does this for the camera location

But I need it for all (well, all tagged) NPC locations

[slate scarab]

i guess the ambisonic encoding would allow you to kinda just get this information

[rapid hedge]

Agreed, was just wondering if this kind of stuff should be in the node or not

Basically I want to run the entire node for each NPC

[slate scarab]

Definitely not as-is -- it would be super expensive to decode the field for every npc. Well I mean you could probably get away with it, but it would be a little wasteful.

[rapid hedge]

Yeah that’s what I thought too

[slate scarab]

It's tricky though -- you can't do the decoding too lazily, or you'll miss short sounds.

or infrequently i mean

[rapid hedge]

That’s fine 🙂

The NPCs only listen every 200 ms

[slate scarab]

well it's tricky from the audio perspective -- you could have a really loud sound, but if it's too short, you'll miss it if you infrequently poll

[rapid hedge]

Yeah I see

[rapid hedge]

In the original, they actually have a queue of sounds to be processed that they then drain when the NPCs listen

But I'm not knowledgeable on the details

[slate scarab]

I don't completely know how the ambisonic encoding works, but you might be able to accumulate the average amplitude for each channel. If you do it this way, you can basically send back nine floats that allow you to "reconstruct" an ambisonic field with that amplitude. Then, you'd decode it with the decoder, and measure the amplitude that their ears receive.

[rapid hedge]

Oof that sounds like math

[slate scarab]

No it would be trivial I think -- just produce a sine wave or something with that amplitude for a few samples haha.

[rapid hedge]

Hehe

[slate scarab]

Alternatively, you could just do the full processing chain in the ECS too haha

[rapid hedge]

That’s what I'm thinking

Since we only really need the realtime stuff for the actual sound on the speakers

[slate scarab]

It's maybe not super efficient, but it would be a near perfect reproduction of what the NPCs perceive (except for maybe any additional effects you have in the audio graph).

[rapid hedge]

It’s fine if it’s fairly rough

For the CPU visibility I just assume everything is a perfectly diffuse white sphere

[slate scarab]

You could probably reduce the cost a ton by running it with a much lower sample rate too

[rapid hedge]

I was also thinking that the node could keep a buffer of its last output

And then the ECS can just read that buffer whenever it feels like it

[slate scarab]

What I would do actually is just use the audio clock + the sample playhead, and then you can know exactly what has played over the previous frame (in terms of a time range within the sample).

[rapid hedge]

Oooooooooh

So then I could read those frames from the sample in the ecs

[slate scarab]

ya

[rapid hedge]

And run the processing in the ecs too

[slate scarab]

(and reduce the rate by like 40x)

[rapid hedge]

I hope steam audio likes that

Given that I pass the sample rate nearly everywhere

[slate scarab]

Oh well for your ECS simulation you'd just choose some lower rate.

And pass that around.

[rapid hedge]

And now the simulator used a different sample rate than the processing

[slate scarab]

It doesn't say you can't have more than one context / simulator.

[rapid hedge]

Fair

[slate scarab]

Although presumably you'd only want to tick the simulator once per frame.

[rapid hedge]

Though I wonder if I can just reuse the existing simulator

[slate scarab]

Yeah I'd hope it doesn't need the sample rate.

[rapid hedge]

Docs say to not even do that FYI

But to run it in a background thread, then check every tick if it was done

(Which makes sense to me)

[slate scarab]

Ya either way, it would be best to avoid running it 2x haha

[rapid hedge]

Hehe

[rapid hedge]

I'll definitely need some API help when I get here haha

[slate scarab]

damn it does require the sample rate on construction

[rapid hedge]

The simulator? Yep

It’s uuuh

Not ideal

Maybe nothing bad happens if I process it with a different sampling rate

[slate scarab]

If you lie about it (like just run ECS simulation much slower anyway), the filters will have a significantly different attenuation profile

which might mess with the amplitude calculations.

[rapid hedge]

Heck

[rapid hedge]

So maybe this could make more sense?

[slate scarab]

i mean it probably wouldn't be that bad if it's only 2 or 4x slower

[rapid hedge]

Then the simulator only ran once, at least

[slate scarab]

well I'm not sure this really avoids the problem, you'll still have to go through a bunch of processing -- the main expensive step here will be the decoding for every NPC

[rapid hedge]

Hmm I see

Well, it’s more like every NPC tries to listen every 200 ms, which is randomly offset per NPC

So I hope it won’t be too many requests per update

Realistically, we are looking at 100 NPCs max per level

But if they’re far enough from the player, the update rate can also go down to 500 ms

[slate scarab]

whew okay but that is quite a lot haha

[rapid hedge]

Yeah but with a bit of luck that calculation may end up at like two active listeners per frame

(Number pulled out of a hat)

[slate scarab]

Yeah I suppose if you're okay with merely probing a short snippet of sound every 200ms, then it wouldn't be that bad.

[rapid hedge]

I'll need to dig into the reference to see how they deal with inter-update sounds

And then come back crying here hehe

Ah btw the draft PR plays sounds now!

Idk why they are buggy

But sounds!!

[slate scarab]

okay I'll see if I can get the channel stuff going here real quick and take a look

[rapid hedge]

Thanks!!!

Definitely need your help to debug the rest haha

Thanks again for your unending patience!!

[rapid hedge]

Just checked and I'm fairly sure the C++ code does this, yeah

Namely checking listening every 200-500 ms for things that are playing right now. Shorter sounds seem to be ignored

Well, usually the levels are indoors

So there’s reverb

So sounds will ring out a while

But yeah if the peak amplitude was between listens, I think it’s not heard

Ooooh

It’s simpler

The sound itself is tagged with a SoundType

And if the volume that ends up at the NPC is at a threshold, it will get a fixed attention according to that tag

e.g. a thrown plate will always generate the same awareness when it was heard, no matter how loud the sound

[slate scarab]

mm, that's a touch more expensive with this approach since you'd presumably have to perform the decoding for each NPC for each sound

otherwise they'll all get mixed up into the sound field

i mean definitely still possible, but hopefully you could make the decoding really cheap

[rapid hedge]

Yep

[rapid hedge]

Hope so too

At least I can use a whole bunch of heuristics. Most sounds that are playing (e.g. mechanical whirring) aren’t relevant for gameplay, so I can ignore those. The only important stuffs are

• player footsteps
• other NPCs broadcasting information
• suspicious actions like lock picks, thrown objects, etc.

[slate scarab]

you could even have a special pool for semantically significant sounds

[rapid hedge]

And for all of those, I can first check distances to the listener

The original code even hard-caps at 4 per frame

4 processed audios per listen per NPC that is

[slate scarab]

ya that's probably a good idea

[rapid hedge]

Yeah then those can get the more complicated node that runs per NPC

(per NPC requesting the listen)

[rapid hedge]

@cold isle my PR https://github.com/MaxenceMaire/audionimbus/pull/20 is ready. I left comments explaining why I did stuffs 🙂

[rapid hedge]

I still need about 300 LOC of wrappers to make it actually compatible with firewheel, mostly due to missing impls on RealtimeClone, Diff, Patch

if you want, I could add a firewheel feature to audionimbus

But this here is also fine

[rapid hedge]

Oh and @drowsy dome you’re also an audio expert, so feel free to help me debug https://github.com/MaxenceMaire/audionimbus-demo/pull/2 if you feel like it :p

[drowsy dome]

I appreciate the mention :) yeah I'm actually looking at this rn because I need to do some things following this chain of dicussion

everyone here: I am an audio beginner. I steal soundfonts and have yet to do anything beyond parsing sf2 and creating a scrappy saw wave

hehe. so onlookers don't ask me for much info beyond midi 😆

[rapid hedge]

Psshhht don’t listen to him he literally maintains a midi library

And made a virtual RAT piano

[drowsy dome]

hehe to be a rat shader 🐀

[rapid hedge]

I heard Daniel is an audio node

That’s how good he is

[drowsy dome]

wait this shit is cool

im forking to look at everything

[slate scarab]

the rat piano is important

[limpid mason]

How have I not heard about this yet?

[rapid hedge]

Would be cooler if the audio played correctly >:[

@slate scarab did you stumble on any low hanging fruits?

[slate scarab]

Not yet. Earlier I finished implementing the automatic up/dowmixing, so you'll be able to just toss them in sample_effects once I push that. Some tests are failing from the Firewheel update though, so I don't know how long that'll take to resolve.

[rapid hedge]

Heck yeah 😄

[drowsy dome]

hmmph macos angry at a lad named phonon

[slate scarab]

ya u gotta link

[drowsy dome]

oh ofc

[rapid hedge]

or use the auto-install feature

[drowsy dome]

im using the auto-install feature actually

but uh, idk. maybe somethin up. im still parsing what's off

[slate scarab]

https://docs.rs/audionimbus/0.8.3/audionimbus/#installation

for manual

[rapid hedge]

[drowsy dome]

ah ha ive found thym issue, which is that the path appears to be different

but im surprised im the first

[rapid hedge]

macOS has no audio devs confirmed

[drowsy dome]

nothin like a bit of a hack in the generate_bindings_phonon part of the audionimbus-sys build script hehe

let path = tmp_dir.join("steam_audio_cache/steamaudio_core/steamaudio/include/phonon.h");

wait

ur not on ur no-alloc branch right

[rapid hedge]

once you get it to run, check the master branch for how it's supposed to sound

[rapid hedge]

I am

[drowsy dome]

interesting. cuz I'm missing a const: audionimbus_sys::STEAMAUDIO_VERSION. I'm wondering if that's added in the build script 🤔

ok honestly...im just gonna actually install it 😆 was fun.

[rapid hedge]

uuuh may be

[rapid hedge]

I did so too 🙂

[drowsy dome]

im honestly confused about how it's working for y'all since steam-audio/core/src/core doesn't appear anywhere for me. that path is non-existent in the steamaudio download or otherwise.

[rapid hedge]

it's a submodule

that also caught me off-guard lol

[rapid hedge]

git submodule update --init --recursive --checkout

[drowsy dome]

I didn't realize, wow haha

[slate scarab]

imagine a world where git chooses the correct default even once

[drowsy dome]

i need to enable submodules by default

oh yeah

that's crisp

the crispy audio is a feature

[rapid hedge]

lobotomy simulation

[drowsy dome]

yeah i can hear a lil buzz. is that the chorus? i'm trynna figure out why it's so beautiful. I noticed that moving out of the radius of one of the nodes, it sounds completely normal

[rapid hedge]

I thought so too

[drowsy dome]

wondering if it has to do with some phase conflict

[rapid hedge]

but it's not quite normal

Since it's supposed to fade with distance

Also, check this line:

|(direct_sample, reflections_sample, reverb_sample)| {
    (direct_sample * GAIN_FACTOR_DIRECT
        + reflections_sample * GAIN_FACTOR_REFLECTIONS
        + reverb_sample * GAIN_FACTOR_REVERB)
        / (GAIN_FACTOR_DIRECT
            + GAIN_FACTOR_REFLECTIONS
            + GAIN_FACTOR_REVERB)
},

By setting these factors to 0, you can selectively enable only one effect at a time

that way I discovered that the direct sample sounds correct, except it doesn't fade with distance or walls

the reflections and reverb are what sounds buzzy

[drowsy dome]

found the first issue

you extend the buffer infinitely

[rapid hedge]

heck yeah

[rapid hedge]

there's a drain

[slate scarab]

i was gonna say check that you clear the buffers haha

[drowsy dome]

this code:

for buff in &mut self.output_buffer {
    buff.extend(std::iter::repeat_n(0.0, input_len));
}

reallocates, draining keeps the allocation

imma see if that deals with the clicking

[rapid hedge]

Are you sure? I copied Corvus' design from here: https://github.com/CorvusPrudens/firewheel-ircam-hrtf/blob/97ffe3e6fa182767be8192bdee4e922fd47e9bc5/src/lib.rs#L366-L373

[drowsy dome]

ok a lot of clicking is gone now: buff.resize(input_len, 0.)

[drowsy dome]

this is right

oh

oh...

[rapid hedge]

[drowsy dome]

mr crow is this doing what I think it's doing https://github.com/CorvusPrudens/firewheel-ircam-hrtf/blob/97ffe3e6fa182767be8192bdee4e922fd47e9bc5/src/lib.rs#L339-L340

[slate scarab]

probably not

it's pre-allocated

[drowsy dome]

hmm. i wonder why that fixed clicking

[rapid hedge]

Just printing the length also tells me it's not growing:

[drowsy dome]

you're right

[rapid hedge]

But I'll be honest

I'm not entirely sure what the crow code does

I get it in principle

[drowsy dome]

yeah yeah ur right mb. I thought it was extending allocation every frame

makes sense hehe

[slate scarab]

(dw i'll be able to get back at it after work)

[rapid hedge]

@slate scarab how come there doesn't need to be a

if self.output_buffer[0].len() < outputs[0].len() {
    return ProcessStatus::ClearAllOutputs;
}

Or is that what the clear_outputs variable is for?

[rapid hedge]

I added that to my code because otherwise I hit cases where the outputs are not fully full

[drowsy dome]

verified though. the capacity is the same. good to know

[rapid hedge]

thx!

[slate scarab]

I think my code isn't super robust here. If you run into this situation, you're screwed either way! There will be a gap.

[rapid hedge]

well I run into it 👀

according to debug printing

[slate scarab]

The proper solution would be to accept some additional latency and only start draining the output when the output buffer is some factor of the size of the max buffer.

It might be fine if it's 1x or more actually.... but to be safe, you'd probably want to start draining when it's at 1.5x or greater.

If you start draining the output eagerly, there may be gaps.

[drowsy dome]

something interesting: return 0. here

[slate scarab]

(If this isn't clear lmk)

[rapid hedge]

Like this?

let max_buffer = outputs[0].len();
if (self.output_buffer[0].len() as f32) < max_buffer as f32 * 1.5 {
    return ProcessStatus::ClearAllOutputs;
}

for (src, dst) in self.output_buffer.iter_mut().zip(outputs.iter_mut()) {
    for (i, out) in src.drain(..max_buffer).enumerate() {
        dst[i] = out;
    }
}
ProcessStatus::outputs_not_silent()

[slate scarab]

Well that's not the max buffer -- you'll have to get it from construction. The actual buffer size of the inputs and outputs is variable.

[rapid hedge]

that's silent for me

[drowsy dome]

oh, it's clicking for me. huh.

[rapid hedge]

but I also played around with the output buffer

[rapid hedge]

oh is it buffer_size.max(FRAME_SIZE)

[drowsy dome]

oh wait what

[slate scarab]

You'll probably also want a flag along the lines of "started producing output," since 1.5x may not be true intermittently.

[slate scarab]

no like you'll want to get the max_frames from the context in AudioNode::construct_processor

[rapid hedge]

max_block_frames?

[slate scarab]

ya

The first time the buffer reaches some factor of that size, then you can start draining.

You can probably exactly calculate what that factor needs to be, something like max_block_frames.div_ceil(fixed_frames)

well, that would tell you how many fixed_frames worth of samples you need before draining anyway

[rapid hedge]

if !self.started_draining {
    if (self.output_buffer[0].len() as f32) < self.max_block_frames.get() as f32 * 1.5 {
        return ProcessStatus::ClearAllOutputs;
    }
    self.started_draining = true;
}

let output_len = outputs[0].len();
for (src, dst) in self.output_buffer.iter_mut().zip(outputs.iter_mut()) {
    for (i, out) in src.drain(..output_len).enumerate() {
        dst[i] = out;
    }
}
ProcessStatus::outputs_not_silent()

like that?

(where I prealloc max_block_frames * 2)

[slate scarab]

ya that should work

We're basically pipelining the audio here, and there's same startup latency.

[rapid hedge]

neat

[slate scarab]

But once that latency passes, we should never fail to fill the output.

[rapid hedge]

unfortunately, the sounds sounds the same as before 😄

But I bet this would have been a problem down the road

so good we tackled it

[slate scarab]

yeah it definitely would have

[rapid hedge]

does this make sense?

I just pulled a factor out of a hat

just to make sure we're bigger than the 1.5

[slate scarab]

I think it's not completely true, but I can revisit it later

[drowsy dome]

rq what is the single input channel actually doing?

[rapid hedge]

I take in a stereo input and average the two channels into a mono

since steam audio wants mono

you may want to git pull

I'm really really confused why the direct channels sound good but completely ignore the camera position

they respect the orientation

I successfully hear the sound source from left or right

but all walls and distances are not factored in

[drowsy dome]

ig...if i just return silent at the start, whomst has filled the output buffers with the sample on output channels 0 and 1

:o

oh

it has a sample player

ok so ive been fucking around. before doing anything, I set channels 0 and 1 all to 0.

in the final loop, I pushed values only if the channel was 1

and I return *direct_sample.

here's the result:

[rapid hedge]

sounds a bit like a cicada

[slate scarab]

hey man cool bug you got there

[drowsy dome]

is there a ref for what these channels mean

[slate scarab]

(which channels?)

[drowsy dome]

the 9 output channels

[rapid hedge]

prolly here somewhere: https://valvesoftware.github.io/steam-audio/doc/capi/guide.html

[steep dove]

that's certainly some audio

[rapid hedge]

pub(crate) const AMBISONICS_NUM_CHANNELS: usize = (AMBISONICS_ORDER + 1).pow(2);

[slate scarab]

that some ambisonic shit
it's an encoding of a "sound field," so there's the concept of pressure zones and stuff

[rapid hedge]

something something spherical harmonics

[slate scarab]

big math words

[rapid hedge]

calculator? why not calc you now

[drowsy dome]

iquick could run a circle around me right about now

well hopefully ill find the cricket

you always hear em chirp but never see em

[rapid hedge]

hecking sneaky

@potent moat you up for helping out with Steam Audio?

[drowsy dome]

it's actually pretty decent with an ambisonic order of 1
edit: no it isnt

[rapid hedge]

sounds the same on my end

[drowsy dome]

hmm

[rapid hedge]

Same

[drowsy dome]

i mean this is a bit dense for me it's denser than me tbh. i've been looking at uh like how sources propagate with this system vs the main branch, and I think I've entered a tailspin in terms of understanding

im wondering if for some reason, it's like setting the inputs every frame or something. like idk.

[rapid hedge]

Relatable

We need the crow

CAW

[slate scarab]

i shall return.... eventually this evening 🤣

luckily i fixed the failing tests i was experiencing earlier so i won't have any blockers

[drowsy dome]

sanity check: mono input would in theory interleave a stereo source right

well doesn't matter. seems the input container used to deinterleave

alright yeah gg, not sure.

[slate scarab]

No, a mono input should sum the stereo source -- that is, every frame is just a sum of the left and right samples.

If I'm understanding you correctly anyway.

[rapid hedge]

Okay, I have a pro gamer idea

I'll just fill the input with uuh [0.5; FRAME_SIZE]

and dump the output into a text file

then I check at which point in the code they diverge

[rapid hedge]

well this is how it starts

that level of divergence seems fine

now let's look how it ends

[hearty bear]

0.5

[faint wigeon]

@rapid hedge loving following along with your entry into audio coding

[rapid hedge]

hehehe

thanks

Seriously, I'm learning sooooo much from Corvus over the last few days 😄

[rapid hedge]

more or less, give or take, meh odr weniger

after the first node

looks fine I think

aha!

this looks like crap!

or

hmm

[slate scarab]

oh boy 😅 i can't wait until we have an editor so we can make scopes and spectrum analyzers

[rapid hedge]

no, actually this is just because the seedling output buffer is smaller than FRAME_SIZE

[rapid hedge]

but no, this doesn't make sense either

[rapid hedge]

I'm debugging like God wanted /j

[potent moat]

Hallo! Whatcha trying to do?

[rapid hedge]

https://github.com/MaxenceMaire/audionimbus-demo/pull/2

Bevy seedling + audionimbus

but it's not doing exactly what it should >:[

well well well

I'm no master mathematician

but - ain't +

[drowsy dome]

im inclined to agree

where's this at? i haven't followed along

honestly looking at raw samples, it's difficult to tell how valuable a diff is. is there some intuition here that can be made?

would have expected the first order derivative diff might hint at something spookier...

[rapid hedge]

this is looking closer

[rapid hedge]

output of the AmbisonicNode

[drowsy dome]

if you df/dt i think that might be useful as well :o

[rapid hedge]

Now bisecting this like a caveman

[slate scarab]

btw it might help to encapsulate the fixed buffer processing -- i can do that in a little bit here

it can very be self-contained

[rapid hedge]

I'm afraid of the old and the new code processing things at different intervals

[drowsy dome]

ah yeah

[limpid mason]

Same, it's a chill blog to read (and educational!)

[rapid hedge]

this is how to transfer data between one process and the next, right?

[slate scarab]

yeah the followup panic is a critical step

[rapid hedge]

Successfully imported some correct intermediary data

[drowsy dome]

hahaha

[rapid hedge]

the decoding node produces almost exactly the same output, given the same input

alright, so I think it's one of the following

• the way outputs from node A are buffered and fed into node B is broken
• the way outputs from node B are buffered and returned is broken
• the Steam Audio API calls are wrong

super secret extra option: audionimbus marked something as Sync that is in fact not Sync

[slate scarab]

oh man i have to set up an ssh key for these steam repos

[rapid hedge]

Oh really?

I can just send you a zip of the submodule if you want

[slate scarab]

no stand back i got this
im a hacker

[slate scarab]

wow that was a huge pain

it worked though

[rapid hedge]

@slate scarab if you fix this

I will literally

idk

send you enough money for a Happy Meal

[slate scarab]

lmao bet

[rapid hedge]

I require sustenance

brb

Guess what happens next

[slate scarab]

no water just break it up a bit and eat it like a cracker

[rapid hedge]

lucky for me, no matter where you live, a Happy Meal can only be cheaper than here

anyhoot

I hope it builds locally for you haha

[slate scarab]

yes
i was hoping everyone was just underrunning because the dev profile had no optimizations

but it's still buzzy with them 😅

[rapid hedge]

the other issue is that when you set the gain of the non-direct audio to 0, the direct audio sounds correct, but has no occlusion or air resistance

also, thanks for that great testing sound file hehe

[slate scarab]

As if it's not being applied at all?

[rapid hedge]

fly out of the room and away

it doesn't go down in volume

and I can't tell any difference between inside the room and outside

in contrast, the reverb and reflections, buzzy though they may be, at least respect the walls

But it's not like the direct sound is just ignoring the camera position. Rotate the camera and the audio will correctly pan from one ear to the other.

[slate scarab]

Oh really? I'm not even observing that oh well probably because I just broke it. I hope you don't mind me using the latest commit from seedling to get the effect in the pool properly. I'm converting the queries to not be Singles now.

[rapid hedge]

oh yes plz

wait, I'll give you collab access

then you can just push

done

https://github.com/janhohenheim/audionimbus-demo/invitations

[slate scarab]

oh i didn't know github had a nice feature for this

oh right, really there should just be one decode node

[rapid hedge]

could have one per NPC, since they have different listener positions

[slate scarab]

ya in that case you'd want an ambisonic bus probably, and you could route all ambisonic pools to it (and then route the bus to a decoder for the actual player)

but i'll keep it simple here

[potent moat]

I could help take a look, but audionimbus is more mature than my old steam audio library

[rapid hedge]

I would be super super super glad if you could double check the API calls

but you'll have to share the Happy Meal with Corvus

Corvus gets the toy though

[slate scarab]

mildly pleased meal

[potent moat]

is the library author in the disc?

[rapid hedge]

ye

but they were already pinged, so I don't want to spam ping 😄

it's @maxence8541

[potent moat]

Gotcha

So the issue you're having rn is direct sound hrtf stuff works fine, but scene stuff doesnt?

or is it the effects ontop of direct

[rapid hedge]

• direct audio sounds fine, but only the rotation of the head is respected. There's no occlusion or distance volume down thingy (air resistance?)
• reverb + reflect sounds buzzy, but correctly respects walls

@slate scarab any leads I can investigate?

I'm completely out of ideas

[slate scarab]

still working out the per-sampler configuration (for some reason it's going silent atm, even though the connections look good in firewheel's debug output)

once that's set up I shold be able to get to the meat of it

[rapid hedge]

tofu you mean

[slate scarab]

sorry i didn't realize this was a vegeterian investigation

[rapid hedge]

vegan

I need to clarify this or I may get my vegan powers removed

[slate scarab]

i am beginning to unravel the tofu

[potent moat]

hmm

I'll have to look at what audionimbus does, distance attenuation worked fine out of box when I made the other library

[slate scarab]

Well the issue with that is that the settings aren't being updated -- I've fixed that bit and the distance / occlusion is working as expected.

oh, same with the orientation

[rapid hedge]

oh heck yeah

thanks!

[slate scarab]

wow the occlusion is really harsh without reflections

[potent moat]

ya, it's just a direct raytrace I believe

[slate scarab]

turn the corner and it silent haha

[potent moat]

there's a lot of types of reflections and reverb too

[rapid hedge]

may I request 1x git push, kind master crow?

[potent moat]

gets really interesting

[slate scarab]

ya just a min, i got a couple things in progress here

[rapid hedge]

@potent moat do you happen to know how pathing is setup?

audionimbus doesn't have a demo for that

but from what I can gleam, the API suggests you have to setup probes manually

is that true?

Do you, like, just spawn them in a 3D grid?

[potent moat]

They don't have scene stuff? Or wat

[rapid hedge]

sure they do 😄

I mean this: https://valvesoftware.github.io/steam-audio/doc/capi/guide.html#pathing

I'm just not sure if this means I can tell Steam Audio to set these probes up for me

[potent moat]

Oh I'm not entirely sure but I think it's manual

[slate scarab]

okay pushed, lmk if you run into any issues getting it building again @rapid hedge

it should just work

[rapid hedge]

yay can confirm occlusion works!!

[potent moat]

Oh ya look a bit up from that portion of the docs jan

[rapid hedge]

let me check the diff

[potent moat]

You specify a matrix of probe position and then put that in an array

[rapid hedge]

OOOOOO

[potent moat]

then you can bake things using that probe array/batch

[rapid hedge]

that's nice!

Will need to try it out once this here works 🙂

thanks!

[potent moat]

ya, tbh I'm certain you could make a smart voxel based autoplacer to get good probes

But that's future smart stuffs

When I was looking into this a while ago I believe that'd what microsofts audio engine did

Honestly if you used something like rerecast you could prob get something similar in good autoplacing

better as a separate library tho lol

[rapid hedge]

@slate scarab how does this work order-wise? Does the signal first go through the effects, then the volume node, then the chained node?

[slate scarab]

Yeah so it might not be super clear from the bundle, but the node on the pool itself is always last. All members of the pool are routed to it. So it's SamplerNode -> AmbisonicNode-> VolumeNode -> Decoder

The docs mention this almost in passing, so I suppose that could be expanded on.

[slate scarab]

oh to be clear *last in the pool, the decoder is outside of the pool, so we're chaining the pool (i.e. the volume node) into it

[rapid hedge]

aaah and I see where I used the wrong reverb_effect_params. But I thought the processor was rebuilt when the node changed. Is that not the case?

Namely I would have expected a changed simulation_outputs to rebuild the processor

[slate scarab]

No, only when the configuration changes. Rebuilding any times the param changes would be quite inefficient! That's why we have the diffing.

[rapid hedge]

ah, gotcha 🙂

[rapid hedge]

Does this already allow doing 3 separate (SomethingNode -> VolumeNode)?

Or does that require more setup

[slate scarab]

no because that would be a graph, not a mere chain

all the items in sample_effects! are chained one into the next

When we flesh out the "subgraph" idea, that can change.

[rapid hedge]

If this were a proper integration library, how would one encode things like the AMBISONICS_ORDER so that they're user-configurable?

Given that all nodes and the simulator depend on a common definition

I guess you could set it up as part of the plugin? Or automatically mutate all nodes once a resource with it changes?

Oh and @potent moat there's no way to change the sampling rate of the simulator without completely rebuilding it, right?

Also I wonder if the audionimbus::Context should be hidden from the user, or proudly exposed so that they can do whatever

[slate scarab]

If the integration is purely forbevy_seedling, I'd use a config struct with a private field. Immediately upon insertion, you could mutate it to match some ambisonic order resource.

If you also mutate it in response to the user adjusting the ambisonic order, it will correctly rebuild all the nodes.

[rapid hedge]

there's something lovely about the idea of "just add this node and it works magically" without having to even know what a context is

But that would require ugly things like hiding a lot of state behind Options because the user could spawn everything before the context even exists

Also the AudionimbusPool looks very neat now, but if behind a library, a user would probably want to configure the different nodes

So I can imagine they would still need to manually spawn the effects

(unless I understood the API wrong)

[slate scarab]

Well really you could do whatever you want. For example, you could expose the important configs as some meta-component that a user inserts on the pool node, and then it handles automatically adding the effects.

[rapid hedge]

Ooooh that's a neat idea

I like that

Once this works I'll basically copy-paste it into the Thief AI to iterate on it, but I can see myself turning it into a little library later 🙂

[slate scarab]

I suppose you could also express the pool creation as an event

[rapid hedge]

But I think it's already great for users if there's just an example for steam audio + seedling at all

Fixed some lints, but if it conflicts with your local changes feel free to force push

[potent moat]

yeah, steam audio takes all the audio settings as const ptrs and I don't think they store them, so you'd have to rebuild hrtfs, effects, etc.

but really sampling rate shouldn't be changing at runtime often

idk if any games even really handle that

[rapid hedge]

Corvus says it happens when switching output devices

[potent moat]

I suppose yeah

[slate scarab]

ya happens easily in that scenario

[rapid hedge]

Which sounds like something I should support 😄

[potent moat]

I don't think you need to rebuild everything though

Scenes and probes and such can be untouched I think

[rapid hedge]

yeah makes sense

[potent moat]

I think it's mainly hrtfs, effects

[rapid hedge]

sources

[potent moat]

Ya

[rapid hedge]

and the nodes by extension

meh

I'm just dreading the bookkeeping

it's not much

but it's boring 😄

dear Steam Audio, just give me a method to change the sampling rate

thanks

[potent moat]

Yeah, would be nice if they handled that for you

I mean MAYBE they do I have no idea, the ptrs are const but who knows if they store the ptrs

[rapid hedge]

hmmm still failing to combat that buzz

did I heck up the output buffer or something like that?

[slate scarab]

oki i fix

pull

[rapid hedge]

wha

HOLY SHIT AAAAA

@drowsy dome check this out

I see you got sick of the fullscreen

(me too)

[slate scarab]

oh sorry haha

[rapid hedge]

no worries that was from Max' original code

[slate scarab]

i don't got time for it slowly moving to another window in macos

[rapid hedge]

ooooh so my very very smart decision to reuse the effect because it was two times the same thing was not as smart

welp

And here I thought an "effect" was something like a pure function

[slate scarab]

You were using the same effect struct for the reverb and reflection, but that's illegal! It was likely rapidly cycling between the different settings, and depending on the kind of things it's doing, it can sound torn up like that.

[rapid hedge]

I see

[slate scarab]

It can happen with the HRTF too even in normal use -- if something moves too quickly across the sound field, it can sound artifacty like that.

[rapid hedge]

yeah in my mind the effect was the config for a pure function to be applied

not something that implied some kind of state

makes sense now 😄

[slate scarab]

ya it's storing state for filters and stuff

[rapid hedge]

heck yeah

this is great!!!

I have some ideas of how to automatically read the topology of TrenchBroom maps into steam audio

should be simple

then we can draw arbitrary halls to test the effects 🙂

aaaaah this is cool!

@cold isle it works now!

[slate scarab]

There's also some optimization opportunities -- for example, we don't need a staging buffer for the decoder, since it already has its buffers arranged as pointers.

[rapid hedge]

@slate scarab set up your GitHub Sponsors and I'll treat you to a meal

no joke

I'm just playing around with the program right now, this is amazing 😄

Thanks a bunch again!!!

[potent moat]

Nice!

[drowsy dome]

wait huh

wait wtf

it was fixed

ok im taking a look i am also sustenencing

where this be

where where

i really wanna see

my brain is tingling

oh wtf it's ont he branch

ok okokokokok

wtf how does fcba530 fix the buzz

im so confused

was it using the reflection effect twice

[slate scarab]

two effect better than one

[drowsy dome]

Shit

[rapid hedge]

yeah turns out the reflection effect implies state bound to it on the steam audio side

[drowsy dome]

But how did you know 😂

[rapid hedge]

so you need the exact same effect twice so it doesn't overwrite its own state

[drowsy dome]

Nice

ok y’all are smarties

[rapid hedge]

(since reverb and reflection are the same thing, just with flipped listeners)

[rapid hedge]

Corvus is

this was all his fixing

[drowsy dome]

Still not sure how that made buzz

[rapid hedge]

it oscillated between two effects

[slate scarab]

It's kinda like if you modulated the parameters of a filter at like 500hz

[rapid hedge]

which in this case would be the equivalent of the camera zooming around all over the place

[slate scarab]

It's cycling fast enough that the oscillation rises into the audible spectrum (or if it automatically handles smoothing, then maybe there are secondary artifacts that contribute).

[drowsy dome]

m

[rapid hedge]

now the important question

[drowsy dome]

That’s lovely I like audio engineering infinitesimally less now

[rapid hedge]

how does the CAW sound

[slate scarab]

real questions

[rapid hedge]

caw sounds pretty decent 👌

[drowsy dome]

can we get a “coo”

[slate scarab]

one thing i thought was interesting is that the binaural setting sounds overly muffled

the high frequency attenuation was pretty strong

but maybe that's partly because the scale is huge or something

[rapid hedge]

is it enough to do binaural: true?

[slate scarab]

ya that's what i mean

[rapid hedge]

oh wow

this tickles my brain

[slate scarab]

idk i assume there isn't other setup required

[rapid hedge]

oooooo

this is neat

I don't hear it muffled

but I don't have very trained ears

it just gives me some goosebumps 👀

some strong placebo they implemented in that feature

[slate scarab]

you might notice it sounding "brighter" if you face one ear directly at the source

[rapid hedge]

oh yeah you're right!

let's see how bright the CAW gets

Now there I hear it kinda muffled

let's set the material to metal

[slate scarab]

it's almost like it's mirrored 😅 it sounds less muffled looking away from the source

(directly away that is, we should expect it to be very clear when it's facing an ear)

[rapid hedge]

metal is kinda hard to hear the difference

but carpet is neat

[drowsy dome]

huh

it's good

finally sounds like room

[slate scarab]

once we've optimized it up, i'd be really interested to see how much we can throw at it

I mean assuming we do a good job, the performance of Firewheel itself will be basically negligible. Really it would be a stress test of steam audio.

[drowsy dome]

ive got an interesting question. using BT headphones (airpods), and playing midi commands from a midi file in my app, it seems like the headphones play nearly directly on time. But, if I press a note with my mouse cursor, or play with a connected MIDI device, the delay is audible.

is this an illusion, or is it some optimization

[rapid hedge]

relatable here with my BT headphones

[drowsy dome]

do u midi

[rapid hedge]

no, but the delay is also audible when setting up game sounds

I even measured it a long time ago because I wanted to make sure I wasn't just imagining things

[slate scarab]

illusion i think -- the clicking allows you to easily perceive the latency

[drowsy dome]

gr

ur probably right, i wish the explanation was cooler

OKOK wait

youtube videos

aha, got ya there

[slate scarab]

they lie

they delay the video a little to match the latency of your audio

[drowsy dome]

how they know

[slate scarab]

you can sometimes catch it doing this if you look closely when you start playing

you can get latency estimates from devices

[drowsy dome]

is there any way to cheat the ear so that you can play a note ahead of time without upsetting the listener

feel like the NES would have answers

[rapid hedge]

feed the audio stream into a predictive AI

[drowsy dome]

ok here's the thing. i have the interesting edge case of knowing what note they will play, but i don't know when

but if I can figure out almost when, i could probably get bluetooth headphones to work

im thinking a quiet anticipatory transient

like a reverse sample

[slate scarab]

hmmm ya maybe

[rapid hedge]

(I'm still so happy about the Steam Audio stuff working )

[slate scarab]

I think with just a little more work we could wrap it up in a crate that's really nice to use. Which I think is super cool!

[drowsy dome]

this work seems to be rerecast related, yeah?

[rapid hedge]

Thief AI related*

[drowsy dome]

oh

[rapid hedge]

But I would do a backend style like in rerecast

where we have another crate for avian integration

I'd do the same here: offer a second crate that automatically generates steam audio meshes from avian colliders

And a builtin backend that does the more naive thing of just turning Mesh3d into steam audio representations

[drowsy dome]

im telling you, everything there and here is moving 3d so far forward for bevy, disappointed ive not been able to use it yet, but its day is coming. I'll be sitting with you on that ol video of that workflow soon enough

but spatial audio here is like

really really cool it's workin

[rapid hedge]

hehe my pleasure

[rapid hedge]

I think Bevy is really lovely to do 3D in, you just have to somehow have the insider knowledge of which crates to use and how to combine them lol

and how to do things like the multicam hack

you know the one

[drowsy dome]

yes, but at the same time, it feels like any game that involves a story with multiple levels is yet difficult. and I still haven't made up my mind if that's because of the nature of ECS.

[slate scarab]

ah yes, the one where you.... hack multiple cameras
yes im familiar fellow 3d devs

[drowsy dome]

i probably won't know the answer to that for a decade

[rapid hedge]

basically a lot of rendering features are only ever tested with one camera, but first person games typically use at least 2, usually 3 cameras. To make them work together without bugs, you need this ugly duckling:

pub(super) fn plugin(app: &mut App) {
    app.add_observer(make_cameras_play_well_together);
}

fn make_cameras_play_well_together(
    add: On<Add, Camera>,
    mut cameras: Query<&mut Camera>,
    mut commands: Commands,
) {
    let entity = add.entity;
    let mut camera = cameras.get_mut(entity).unwrap();
    if camera.order == isize::from(CameraOrder::World) {
        // Use the world model camera to determine tonemapping.
        return;
    }
    // Needed because of https://github.com/bevyengine/bevy/issues/18902
    commands.entity(entity).insert(Tonemapping::None);
    // Needed because of https://github.com/bevyengine/bevy/issues/18901
    // and https://github.com/bevyengine/bevy/issues/18903
    camera.clear_color = ClearColorConfig::Custom(Color::NONE);
    camera.output_mode = CameraOutputMode::Write {
        blend_state: Some(BlendState::ALPHA_BLENDING),
        clear_color: ClearColorConfig::None,
    };
}

[slate scarab]

haha that's a lot of issues

[rapid hedge]

@faint wigeon hates this too haha

[drowsy dome]

someday

someone should make a 3d spectrogram for the phase

if possible. idk, im saying things ofc

(useful for chorus i imagine) (again idk if that's actually a thing either)

[slate scarab]

it would probably look blobby

maybe we could make a little 3d analyzer 🤣 imagine

[drowsy dome]

:o

[slate scarab]

we’ll have an audio pane in the editor and we’ll put that one over in the corner

you can spin it around like a little cube

[drowsy dome]

wrote a shader this weekend right, it had that uh, perlin noise thing. but i moved my cube around and like, i was getting slices of the noise on the face of the cube as i moved it through space. kinda cool

would be cool to make "shadergram"

[drowsy dome]

lol yeah

[rapid hedge]

set it up for VR

[drowsy dome]

wait perlin noise reminds me a lot of uh that table trick

hm.

[rapid hedge]

Is it intentional / known that #[require(...)] doesn't work with pools?

[slate scarab]

Yeah, there's probably a better mechanism to make it work nicely there. The issue is that it needs a hook to insert a "type-erased" label. But I don't want to reimplement the entire Component derive macro.

Maybe there's a better solution lying around, but it certainly hasn't struck me yet.

I suppose SamplerPool<L> could register an observer for L then the only failure case is if you never spawn a pool with that label.... in which case your samples will get re-routed to the default pool if possible.

Oh but then you must spawn the pool first. Which isn't so ideal.

[rapid hedge]

meh, I see

[slate scarab]

The alternative, of course, is that you manually register your pool labels. But I despise manual registration.

[rapid hedge]

it's fine though 🙂

[rapid hedge]

I was wondering if I should make an own node for the NPCs. In that case, how do I tell seedling to not route it to the main bus?

Given that the audio should never actually play over the speakers in that case

[slate scarab]

just make a node with no outputs

[rapid hedge]

clever!

[rapid hedge]

@cold isle how do you feel about adding an optional firewheel feature to audionimbus?

For adding additional derives needed for use in nodes

If that sounds bad, I would probably use a fork of audionimbus for my own projects to not have to maintain the 300 LOC of trivial wrappers

But that’s fine too 🙂

[cold isle]

Sounds good to me 🙂

[rapid hedge]

Great, thanks 😄

[cold isle]

Awesome work!!

How helpful do you think a bevy_seedling<>audionimbus plugin would have been for you?

I suppose it would save the effort of implementing components

[rapid hedge]

Well it would have saved me all work I had to do until now haha

[cold isle]

although I'm wondering to what extent they can be generalized to support any effect?

[rapid hedge]

We discussed this a bit

Should be doable to create a node for each effect

And then configure those nodes as needed

[cold isle]

That'd be neat 🙂

[rapid hedge]

Or use a single node with a config component

It’s a pretty good match, API wise!

[cold isle]

Can't wait to play Thief remastered in Rust 😉

[rapid hedge]

@cold isle mind if I replace some usize with u32? Given that they're cast as i32 down the line anyways, this seems fine

[rapid hedge]

I'm doing this because firewheel prefers concrete sizes

@dusky mirage this is about usize not implementing Diff / Patch

[rapid hedge]

actually, given that all values are in the end cast, it seems good to do this anyways

since usize is currently lying to the user about audionimbus' capacities

[rapid hedge]

Hmmm

When I change

firewheel = { version = "0.8.0-rc.1", default-features = false, optional = true }

to

firewheel = { version = "0.8.0-rc.1", default-features = false, optional = true, features = ["std"] }

bevy_ crates start appearing in my Cargo.lock

I don't like that

[slate scarab]

Hm, it's not a direct dependency of any firewheel crates

the only bevy dependencies are subcrates

[rapid hedge]

bevy_platform makes sense to me

the others not so much

[slate scarab]

probably accidental activations (bevy_ecs/std instead of bevy_ecs?/std)

[rapid hedge]

https://github.com/BillyDM/Firewheel/issues/80

[rapid hedge]

sounds like it, but I couldn't find it at a glance

[slate scarab]

hopefully it's not transient

*transitive

ya idk looks well behaved in the main and subcrates

[rapid hedge]

¯_(ツ)_/¯

[cold isle]

Yes, good idea 👍 in fact I should have used u32 anywhere it's not used for indexing

[rapid hedge]

done

Now the sync thingy is left

This can totally be done on the firewheel side, it's just a bit annoying

Need to create a wrapper for ReflectionEffectParams that looks the same, except for an ArcGc<OwnedGc<ReflectionEffectIR>>

[slate scarab]

hm, does the effect need to be sync? or is that just the impulse response for the effect

but yeah if it really is sync it's easier in multiple ways

easier for the ECS too

[rapid hedge]

I need to give the node access to ReflectionEffectParams AFAIK

Used here in the processor:

And the derives yell at me if the node is not Sync

[slate scarab]

well it wouldn't work as a component either if it's not sync 😅

[rapid hedge]

@slate scarab I could also wrap the entire simulation output in an ArcGc<OwnedGc<T>>

Well, an Option<ArcGc<OwnedGc<T>>> because it's late-initialized

[slate scarab]

ya not super ideal

[rapid hedge]

Probably better to do the OwnedGc dance just for that one reflection effect then

Though I wonder

If I clone that pointer

And put the clone behind an OwnedGc

that's not exactly in the spirit of Sync lol

[slate scarab]

the proper thing there would probably be to manually derive diff and patch for it

[rapid hedge]

Though I drop the original reference right after

[rapid hedge]

I don't feel up to the task for that

[slate scarab]

it’s actually super simple

but i can take a look later

[rapid hedge]

By it you mean ReflectionEffectParams I presume?

[slate scarab]

either that or a wrapper around the IR

[rapid hedge]

but then it's still not Sync -> no component

[slate scarab]

yaaaaaaa

[rapid hedge]

Actually I would be grateful if you could take a look at https://github.com/MaxenceMaire/audionimbus/pull/20 in general 🙂

[slate scarab]

have we checked the source? im trying to find the c file to see if its sync

[rapid hedge]

@cold isle said it's not

[slate scarab]

oh no 😅

[rapid hedge]

at least I think so, according to this

[slate scarab]

but it is send?

[rapid hedge]

should be

[slate scarab]

oh but i guess if you clone it that kinda violates the non syncness

[rapid hedge]

bingo

which is precisely what I do right now

but uuh

it's a bit sketchy, isn't it

[slate scarab]

i have some ideas, ill explore them later

basically: RwLock could work in the ECS — remove it when diffing and place it in a OwnedGc so it can be sent to audio

jank, but might work

actually i have better ideas

[rapid hedge]

Can I leave that to you then? 😄

[slate scarab]

send it immediately upon producing the value as a raw event — dont bother with storing it in the params for diffing

much better

[rapid hedge]

Other than this one thing, the PR is ready from my side

both the audionimbus and audionimbus-demo PRs

[rapid hedge]

I'll add you as a collaborator to the audionimbus fork too, sec

Click plz: https://github.com/janhohenheim/audionimbus/invitations

it's the no-alloc branch

[slate scarab]

oki i will try to find a nice solution after work

[rapid hedge]

Feel free to change whatever on the audionimbus and the audionimbus-demo PR 🙂

[rapid hedge]

thanks a bunch

[rapid hedge]

@cold isle are you sure IPLReflectionEffectIR is not Sync? It looks to me like some buffers filled by the simulation when spitting out outputs, and AFAIK they're read-only everywhere

[slate scarab]

dw if it isn’t m, i think i cracked the case

1min

[rapid hedge]

Baaah nvm, iplReflectionEffectApply definitely mutates it

do not Sync this

in fact, the Rust API should maybe even go so far as to Mutex it

Or at least the .apply methods should take in a mutable reference to its params to make clear what's happening

[cold isle]

To be honest, the more I look at the C code, the less I'm sure 😅 IPLReflectionEffectIR is just supposed to be a handle while the actual impulse response is handled by SA internally. Passing the pointer around should be fine but accessing the underlying object simultaneously is unsafe

[rapid hedge]

Also, the params should not impl Clone

because they're cloning the pointer that way

But then they're no longer RealtimeClone either

eek

[slate scarab]

Okay so I think the problem becomes a lot clearer if we step back a moment and consider what should actually be in the seedling params. Right now, we have a few inputs for the simulation and processing (like source position and listener position) as well as the results of the simulation. But I don't think that really makes sense. After all, the simulation outputs are basically useless to anyone but the audio processor. Even for your AI work, it's not like those outputs are useful. They perform nontrivial transformations on the audio.

Instead, the params should just contain just the inputs, including things like DirectSimulationParameters, which is currently hard coded. Then, rather then awkwardly playing hokey pokey with the simulation results, the system that calculates them should immediately queue them to be sent to the audio processor.

[rapid hedge]

oh that's possible? 👀

that sounds wayyyyyy better

and yeah the AI definitely also doesn't care about the sim outputs

[slate scarab]

The Custom variant doesn't have a Sync requirement, and it's not Clone or anything so it's legitimately not required.

[rapid hedge]

I suppose I can read them from the events: &mut ProcEvents?

do they get cleared every frame?

[slate scarab]

With this setup, there's no need for the params to implement Diff and Patch, so the only wrapping you'd need to do would be to make sure the inputs are actually Send and Sync (some of them aren't).

[slate scarab]

not exactly -- more like: if they don't have a timestamp, they'll be cleared after the audio processor has been called with the opportunity to read it, and if they do have a timestamp, they won't be cleared until their time is passed

[rapid hedge]

hmm so if I only run the simulation every 200 ms

do I write the old outputs into the events every frame?

oh wait

[slate scarab]

no the processor could just use the old ones itself

[rapid hedge]

that's what the timestamp is for I suppose?

[rapid hedge]

oh ok

so read the events, if there are newer ones, replace the ones the processor is storing

got it

[slate scarab]

ya

they'd still have to be Options since the audio procesors won't be initialized with valid simulation results, so it should just work out

in other words, if the audio processor hasn't received any simulation results ever, it should just skip processing

[rapid hedge]

with this approach, the outputs don't need to be diff + patch either, right?

[slate scarab]

right

[rapid hedge]

Cool!

[slate scarab]

they just need to be Send + 'static

[rapid hedge]

that they should be

I'll definitely need your help to implement diff + patch for the inputs though

[slate scarab]

they don't need it

there's nothing to diff with this model

[rapid hedge]

oh we should just skip them then?

[slate scarab]

no they don't need to implement those traits at all (you can register_simple_node)

there's probably a better name for that method but

[rapid hedge]

but it still needs the position of the source and listener

[slate scarab]

mm ya i suppose that would still be useful

so yeah you could skipp everything that's not the inputs

[rapid hedge]

makes sense

then audionimbus maybe doesn't need a firewheel feature after all

Hmm the inputs still need to be RealtimeClone though, right?

Which implies Clone

[slate scarab]

i don't think so

Also keep in mind you don't need to store the params struct in the audio processor if you don't want to. With this new approach they'd have a ton of inputs which aren't useful to the processor. You can match on the events directly.

The params struct does need to be Clone though for seedling, so there might be some proxying/transformation involved. Hopefully a minimal amount though.

Like for something like this https://docs.rs/audionimbus/latest/audionimbus/model/distance_attenuation/enum.DistanceAttenuationModel.html, since we can't store the callback in the ECS as-is, we might just make a simpler enum that implements Into<DistanceAttenuationModel>. A touch annoying, but I think that makes sense for an integration library.

Or maybe we could adjust it to have a more rusty API anyway

Seems like that would make more sense as a Box<dyn FnMut(f32) -> f32 + Send + Sync> or something

(or maybe just Arc<dyn Fn(f32) -> f32 + Send + Sync>... more restrictive, sure, but then it would be Clone )

[rapid hedge]

No idea how to bring that over the FFI barrier lol

I'll uh

Leave that for a follow up

[slate scarab]

nah should be easy, just put the Arc<dyn ...> in the void pointer

[rapid hedge]

But the C code wants a function

Not an Arc to a function

[slate scarab]

ya so always give it a function that just calls the closure

[rapid hedge]

Fair enough

[rapid hedge]

I'll still do that though to not lose myself in too deep of a rabbit hole haha

[rapid hedge]

How do i decide what does in the node and what goes in the config component? Could the audionimbus simulation inputs also go in the config?

[tropic bison]

hey, just learned about firewheel from the 0.17 release post and curious if firewheel is already a decent replacement for bevy audio or is it better to wait still before switching?

in my case it’s for a relatively simple 2d game, don’t need fancy spatial audio features etc

[dusky mirage]

Ah yep, I'll patch that real quick. https://github.com/BillyDM/Firewheel/blob/c997f830163ae353a7ec43607b6dd84347f0391a/crates/firewheel-core/Cargo.toml#L45

[slate scarab]

well, but bevy_platform is an unconditional dep!

it shouldn't be causing this issue

i couldn't tell why it's happening either

[slate scarab]

Changing the config will cause the audio processor to be re-created and re-inserted into the graph, so you should generally put stuff in there that you don't expect to change often, or things that require costly operations or allocations like the ambisonic order.

[dusky mirage]

Oh yeah, right. Nevermind.

Maybe try the ol' cargo clean?

And cargo update?

[slate scarab]

By this point I'd consider it an excellent replacement! It has feature parity with bevy_audio, along with a ton of new functionality. If you happen to target Wasm, you can also get multi-threaded audio, which resolves the dreaded cracklies.

[dusky mirage]

Hmm... https://github.com/BillyDM/Firewheel/actions/runs/18335988380/job/52220198177?pr=75

[limber kernel]

did i break something

[slate scarab]

Even if you don't need any spatial features, it's more efficient than bevy_audio (2-3x faster under normal conditions) and makes it easy to add things like reverb, fade ins / fade outs, and more advanced musical sequencing.

[dusky mirage]

No, I think it's just an update to the Rust compiler or something.

@limber kernel Oh yeah, because the impulse response isn't being diffed/patched at all, the ArcGc<()> is not necessary. https://github.com/piedoom/Firewheel/blob/b1d0cfbb099e83a9774557366f7ff9f455868795/crates/firewheel-nodes/src/convolution.rs#L62

[rapid hedge]

Yes 100% ready now imo

[dusky mirage]

That being said, it would be nice to keep the data-driven API by having an impulse response parameter.
You can manually implement Diff for a custom ImpulseResponse parameter like this:

pub struct ImpulseResponse {
    pub impulse: Option<ArcGc<dyn SampleResourceF32>>,
}

impl Diff for ImpulseResponse {
    fn diff<E: EventQueue>(&self, baseline: &Self, path: PathBuilder, event_queue: &mut E) {
        if self.impulse != baseline.impulse {
            // .. construct `FftSampler` and push it to the event_queue
        }
    }
}

However, for this to work I think we need to add a #[patch_skip] field attribute to the Patch derive macro to skip deriving patch for a field. @slate scarab

[slate scarab]

ya we have that

[dusky mirage]

Oh we do? Cool!

[slate scarab]

well, it skips both that is

Oh but I guess if you manually derive one, that's fine.

(#[diff(skip)] is the attribute -- skips diffing and patching if one or both is derived)

[dusky mirage]

Yeah, we need the impulse response parameter to derive Diff, but not Patch.

[slate scarab]

If you want it to be more granular, then yeah we'll have to updated it actually

we could have skip, skip_diff, and skip_patch

which is similar to how serde does it

[dusky mirage]

Or wait, I mean that the impulse repsonse parameter will manually derive Diff, but we also want it to not automatically derive Patch.

[slate scarab]

Right so in that case it should work as-is.