Better Audio

[rapid hedge]

"This specific method with these inputs is safe to call with this other"

"But only if the inputs look like so"

¯_(ツ)_/¯

[slate scarab]

sounds like how people talk about the linux code base 😅

[rapid hedge]

lol I imagine

But the API looks pretty neat now, doesn't it? 🙂

I just want to get rid of SteamAudioSamplePlayer and I'm happy

and add pathing and avian and trenchbroom support

and docs

then we're done

no callback support, no baking serialization

(Except if I feel like it)

I might actually be interested in the serialization

okay, baking serialization can stay, as a treat

BTW RustRover is godsent for debugging the segfaults

[slate scarab]

the callbacks definitely seem pretty niche

[slate scarab]

oh really?

[rapid hedge]

One of the few times I need an actual actual debugger, yeah

Since the terminal doesn't tell me more than just "segfault lol"

and it happens inside the libphonon.so, so no print debugging

[slate scarab]

ya that's tough

[slate scarab]

maybe it's time to try out rust rover ive seen a few people recommend it here

[rapid hedge]

since this is already compiler-optimized, LLDB sometimes fails to fetch the values of variables. But it works for quite a few, which I find impressive 🙂

[rapid hedge]

It doesn't correctly import stuff in this project

I'm just tabbing into it for the debugger haha

[rapid hedge]

(this compiles fine and generates no warnings with rustc, it's just their language server)

It's so weird how many places in the steam-audio library itself just... hope you did everything right?

Like, you create a probe batch but don't put any probes in it

boom segfault

[potent moat]

that's just c for u

lol

I've been tempted to try to do a full port of steam audio now that it's open source

[slate scarab]

not even an assert though? seems crazy haha

[potent moat]

is audio nimbus trying to be just a thin wrapper? Or wrap it with type safety?

[slate scarab]

it seems fairly thin, at least in its current iteration

[rapid hedge]

maybe there are asserts in the debug mode. Remember they distribute precompiled .sos

[rapid hedge]

half half

[potent moat]

honestly I might take a look later and see how big the library actually is

would be nice to have a quality pure rust hrtf audio solution

[rapid hedge]

look how nicely LLDB in RustRover can switch from the C++ calls to the Rust calls up in the stack

it's sooo goood

[rapid hedge]

We have HRTF 😄

It's the spatial simulation part we don't have

[slate scarab]

oh we do have HRTFs in bevy_seedling already actually :) but they're just HRTFs, no occlusion, reflection, or reverb simulation like steam audio can do

[rapid hedge]

that would be poggers

[potent moat]

O nice

[slate scarab]

maybe folks will find bevy_steam_audio so nice and useful that we'll get more momentum on porting the whole thing to Rust

[potent moat]

what hrtf did you guys impl as the default?

Steam audio has one of the best defaults imo

or well where'd you get the hrtf can't really describe it lol

[slate scarab]

you might be aware of this already, but at least one or two other people have started the process https://docs.rs/phonon/latest/phonon/

[potent moat]

Ooo nice

I worked on the steam audio stuff before they open sourced it all the way and been a bit in the dark beside that

Well not steam audio itself the wrapper

[slate scarab]

The HRTF implementation actually comes from Fyrox. Its data comes from the ircam database, with subject 40 as the default. Apparently people like 40. (IRC_1040 to be exact.)

oh i didn't realize it was in a partially open sourced state at some point

[potent moat]

ya it was just the api headers + an sdk until like a year ago or so

[slate scarab]

The Fyrox implementation doesn't deal with the SOFA format like Steam Audio, so we can't just slot in their subjects. It's quite the task to parse SOFA as well, so that would be a big thing we have to tackle for a Rust impl.

[rapid hedge]

it's sooooo cool they opened it. I would not have been able to write the library otherwise, since I absolutely needed the source to know

• what the fuck was going on with the segfaults
• which things are thread safe
• how the Unity plugin works

[sturdy prawn]

mayyybe, though it'd be nice to have some first-party non-Parry, non-rendering mesh type first

I guess it can just return a vertex buffer and index buffer though

[potent moat]

Ya I burnt out on the wrapper because of it because it was a black box

[rapid hedge]

yaaaaay

[potent moat]

I had to report some bugs too, which they've fixed luckily

[rapid hedge]

Huh, I thought it was dead-ish

but that's just the main branch

look at this

[potent moat]

o it's gitghillie

[slate scarab]

i think nth contributed a bit too

[sturdy prawn]

right this was why I felt like the Collider-to-mesh thing is sketchy https://discordapp.com/channels/691052431525675048/1124043933886976171/1385639694426968095

I think the configuration thing might be the biggest problem, since different shapes may want different options

ex: Capsule has two subdivision parameters, one for theta and one for phi

while polytopes have none

in a generic setting, I feel like this may require a builder approach that is vaguely similar to what Bevy does for mesh builders

i.e. there's a Trimeshable trait with some associated Builder type, which can build the mesh, either with default options or manually specified ones

configuring would require you to cast the generic mesh builder to a specific one, e.g. collider.to_trimesh().as_capsule().unwrap().subdivisions(3, 2)
(here, as_capsule would return some Option<CapsuleTrimeshBuilder>)

Mm though I guess I would rather just do collider.shape().as_capsule().unwrap().to_trimesh(3, 2) for that. So to_trimesh should just use some good default parameters, no need for builders I guess

(anyway that's probably off-topic here)

[rapid hedge]

The issue is that I truly just need some config that works with an unknown collider, and I don’t want to write the boilerplate to go through each possible collider. I really just want "plz give me a low-cost approximation of that shape as a trimesh, thx"

I also needed just that for creating debug meshes for colliders in another project. I get that the "right" way is to check each collider and decide on a case-by-case, but in reality, that’s just a lot of boilerplate for very little gain

[rapid hedge]

Also guilty, sorry Corvus and Billy 😄

[dusky mirage]

Wow! Maybe I should look into RustRover.

[rapid hedge]

@slate scarab I merged my pathing stuff to main now that it kinda works. As in, it produces sound and does not segfault. Will need to play around with the processor to make it work better, as its current settings make it a de-facto shitty reverb. I know how to do that though 🙂 Very grateful for the Unity implementation as a template

I just hope I didn't cause you too many merge conflicts!

If I did, I can help iron them out!!

[past sandal]

Ya I just moved into a new place and got a new job (rust job ftw) so I'm slowly starting to get some free time again to work on this stuff 🙂

[slate scarab]

btw @rapid hedge would you still like FromWorld for the configs? That in combination with the new ProcStore should remove the need to LazyLock the steam audio context.

Sorry, actually it's not super easy to design around in bevy_seedling. Default is a trait bound on the associated type directly in Firewheel. Now that I'm thinking about it, though, we should be able to relax that bound.

It's primarily for convenience, but it could be epxressed directly on the convenient methods of the audio context rather than globally.

As we've observed, it doesn't always make sense for the configuration to be Default. Indeed, if some configurations need additional context from the world, a FromWorld implementation is a rather elegant way of expressing that. It could help remove some of the awkward dancing you have to do right now.

In fact, it may be even better if we create our own trait, something like

trait FromEntity {
    fn from_entity(entity: EntityWorldMut) -> Self;
}

impl<T: Default> FromEntity for T {
    fn from_entity(_: EntityWorldMut) -> Self {
        T::default()
    }
}

It doesn't necessarily unblock every use case, but it would allow use to inject arbitrary configuration data from the ECS, without perturbing anything upstream in Firewheel.

Maybe EntityWorldMut is too much of a lie, though. Just &mut World might be better, since I'd like to be able to copy configurations to and from arbitrary entities. So the exact entity on which the configuration is constructed shouldn't really be an API invariant.

[rapid hedge]

Could it be DeferredWorld?

[slate scarab]

that or &mut World is fine -- I'm just imagining we'd insert it with an observer if it doesn't already exist (required components at home), and we could queue that operation

DeferredWorld would probably be faster though (you can get that as a system param right?)

[rapid hedge]

IIRC yes, since it’s readonly

So it should be parallelized with other readonly systems

Btw did you finish your refactor?

I'd like to touch the same files, but I want to wait for you so you don’t get conflicts 🙂

[slate scarab]

I believe the only thing left is to analyze silence for the encoder

Let me push what I have then

I don't think I've introduced any bugs, but maybe keep an eye out

[rapid hedge]

Unity has a fun comment for that, I'll send it later

It’s like "Unity likes to call us with garbage inputs before actually playing anything, so first look for silence before proceeding, as that is the sign that the audio is done initializing"

[slate scarab]

oh boy there are a lot of conflicts for encoder.rs 😅 let me check the diff on main's last commit

oh that's the pathing stuff

[rapid hedge]

You should be able to reproduce my changes

It’s not much

[slate scarab]

oh it's just another effect

[rapid hedge]

Yep, and a bool to say whether there is pathing data available

Because it segfaults otherwise

The "apply effect" part is gated by that

[slate scarab]

Is there an example that uses that effect?

okay i merged that in @rapid hedge, at this point i'll just modify small parts of encoder.rs so it should be fine regardless of what you modify

[slate scarab]

oh i got a pretty good commit hash
babe001

anyway I added those silence optimizations, so the performance should be more or less optimal

[rapid hedge]

Just set all gains to zero and pathing to 1 in the gltf example

[rapid hedge]

Daaaaamn

[rapid hedge]

It sounds a bit like a shitty reverb until I fix the speaker config

But you should clearly hear the music become louder when you step into the sample player sphere

There’s a steam audio builtin to mix buffers FYI

not that we need it

But I noticed accumulate_in_output

Unity does the same, but they keep a steam audio output buffer

And then use that for the DSP output in the end

[slate scarab]

ya, although that mix method requires us to pack the output slices into the C-compatible format, which is kinda annoying

as-is we get to skip that step

(and apply the gain at the same time)

Okay well the performance seems really bad with the default settings 😅 looks like it's time to do some investigating

[slate scarab]

Okay so with 8 emitters using the default settings, it's somehow taking almost 20ms to run the (expensive part of the) simulation. How is that possible 👀

It's also already taking almost the whole audio processing duration

We should be able to support hundreds of sources, so maybe the default settings here are just obscenely expensive.

Interestingly, the audio processing is about 3x faster if the fixed size matches the block size (it was 256, I set it to 1024). Unless something we've done is catastrophically wrong, there must be a pretty high fixed cost to running the effects processing. So we'll probably want to match the native block size if possible. Firewheel's default choice is 1024, so we might set it to that by default at least.

[rapid hedge]

Ouch

[rapid hedge]

The most expensive part is the run_reflection call, or at least it’s supposed to be. But that runs on another thread to not interfere

[rapid hedge]

I can check Unity’s and Unreal's defaults.

[slate scarab]

At least on my machine, tracy's reporting 18ms within a single frame for every expensive simulation period. I guess I should break out perf or something

[rapid hedge]

what is expensive simulation period?

[slate scarab]

200ms

[rapid hedge]

like, which calls? 😄

[slate scarab]

the timer you've set up

[rapid hedge]

aaah

but that shouldn't block

Since it's on an own thread

Oh wait

it would

since I call .write() on the simulator RwLock 👀

I think

or do I

[oak walrus]

Vsauce music starts playing

[rapid hedge]

hehehehehe

Where... are your fingers?

[slate scarab]

i see a // todo: only do this when needed

xD

[rapid hedge]

the synchro part should only be true when the simulation is not blocking

[slate scarab]

oh

[oak walrus]

https://www.youtube.com/watch?v=TN25ghkfgQA

[rapid hedge]

so IMO this call could just as well be try_write().unwrap()

[rapid hedge]

That comment is for the scene commit 🙂

idk if it's expensive to commit the scene if nothing changed

Unity has a flag that means "did the scene change this frame" and gates the commit behind it

but that might just be to make doubly-sure they are not running into thread safety issues, since the commit cannot happen at the same time as the simulation

Oh, also, we don't use any raytracer acceleration

[slate scarab]

oh apparently you can offload some of that to the gpu

[rapid hedge]

Yeah I think it's called Radeon Rays

[slate scarab]

including actual convolution reverb processing, which is neat

(with the unity integration at least)

[rapid hedge]

which works over OpenCL

[slate scarab]

Radeon Rays support in Steam Audio is available on Windows 64-bit only

wtf

[rapid hedge]

wtf wtf wtf

okay, fuck that then

[slate scarab]

no acceleration for me >:[

[rapid hedge]

we go common denominator here

[rapid hedge]

where did you see that?

[slate scarab]

https://valvesoftware.github.io/steam-audio/doc/unity/guide.html#enable-gpu-acceleration

[slate scarab]

might be a unity thing

[rapid hedge]

well here's the C++ docs

Embree sounds nice though

[slate scarab]

can we just use that?

[rapid hedge]

¯_(ツ)_/¯

here's a batch of defaults

Another:

void reset(UnityAudioEffectState* state)
{
    assert(state);

    auto effect = state->GetEffectData<State>();
    if (!effect)
        return;

    effect->inputStarted = false;

    effect->directBinaural = true;
    effect->perspectiveCorrection = false;
    effect->applyDistanceAttenuation = true;
    effect->applyAirAbsorption = false;
    effect->applyDirectivity = false;
    effect->applyOcclusion = false;
    effect->applyTransmission = false;
    effect->applyReflections = false;
    effect->applyPathing = false;
    effect->hrtfInterpolation = IPL_HRTFINTERPOLATION_NEAREST;
    effect->useDistanceAttenuationCurve = true;
    effect->distanceAttenuation = 1.0f;
    effect->distanceAttenuationCurveValue = 1.0f;
    effect->airAbsorption[0] = 1.0f;
    effect->airAbsorption[1] = 1.0f;
    effect->airAbsorption[2] = 1.0f;
    effect->airAbsorptionUserDefined = false;
    effect->directivity = 1.0f;
    effect->dipoleWeight = 0.0f;
    effect->dipolePower = 0.0f;
    effect->directivityUserDefined = false;
    effect->occlusion = 1.0f;
    effect->transmissionType = IPL_TRANSMISSIONTYPE_FREQINDEPENDENT;
    effect->transmission[0] = 1.0f;
    effect->transmission[1] = 1.0f;
    effect->transmission[2] = 1.0f;
    effect->directMixLevel = 1.0f;
    effect->reflectionsBinaural = false;
    effect->reflectionsMixLevel = 1.0f;
    effect->pathingMixLevel = 1.0f;
    effect->pathingBinaural = false;
    effect->pathingNormalizeEQ = false;

    iplSourceRelease(&effect->simulationSource[0]);
    iplSourceRelease(&effect->simulationSource[1]);
    effect->newSimulationSourceWritten = false;

    effect->prevDirectMixLevel = 0.0f;
    effect->prevReflectionsMixLevel = 0.0f;
    effect->prevPathingMixLevel = 0.0f;
}

And the last for now

[rapid hedge]

This suggests that basic stuff like even occlusion are off by default?

Which I am not a fan of

[slate scarab]

One of those sets has the ambisonic order at 1 by default for reflections. Setting it to one allows 32 emitters to reliably run at on the audio thread

Which, you know, makes sense since it's 33% of the number of samples to process

I imagine people would have just a handful of order 2 processors for important things like character voices, and then most sound effects would get 1 (at least if you're processing reflections).

so like a high quality and low quality pool

I guess that's partly why these integrations often set the max number of emitters at like 32 for realtime reflections.

[rapid hedge]

yeah right now max and actual are always the same number

but it could easily be set per-source

[slate scarab]

Well tbf we now skip (audio) processing on any that aren't actually playing.

[rapid hedge]

We can also use all threads for multithreading

[slate scarab]

If I comment out the reflection effect in the audio processor I can run 128 emitters

I guess the reflections are just really expensive in both simulation and audio processing.

[rapid hedge]

dear Unreal

what does this mean

RealTimeCPUCoresPercentage(5)

that's their default value

[slate scarab]

ya idk what that is either 😅 i saw that in one of the other things above

[rapid hedge]

I guess 50%?????

[slate scarab]

or actually just 5%

maybe so audio simulation doesn't slow everything else down

[rapid hedge]

but that would mean you get 2 cores when your CPU has 40 cores total

Some Unreal defaults coming in:

    , RealTimeRays(4096)
    , RealTimeBounces(4)
    , RealTimeDuration(1.0f)
    , RealTimeAmbisonicOrder(1)
    , RealTimeMaxSources(32)
    , RealTimeCPUCoresPercentage(5)
    , RealTimeIrradianceMinDistance(1.0f)

    : bSimulateOcclusion(false)
    , OcclusionType(EOcclusionType::RAYCAST)
    , OcclusionRadius(1.0f)
    , OcclusionSamples(16)
    , OcclusionValue(1.0f)
    , bSimulateTransmission(false)
    , TransmissionLowValue(1.0f)
    , TransmissionMidValue(1.0f)
    , TransmissionHighValue(1.0f)
    , MaxTransmissionSurfaces(1)
    , bSimulateReflections(false)
    , ReflectionsType(EReflectionSimulationType::REALTIME)
    , CurrentBakedSource(nullptr)
    , bSimulatePathing(false)
    , PathingProbeBatch(nullptr)
    , bPathValidation(true)
    , bFindAlternatePaths(true)
    , Source(nullptr)
    , Simulator(nullptr)
    , AudioEngineSource(nullptr)

also no occlusion by default 👀

[slate scarab]

mm they choose order 1 too

if i remove both reflection and reverb calculations (so just simulating direct effects and HRTF spatialization now), i could probably run like 300-400 emitters based on these numbers.

[rapid hedge]

yeah but at that point you don't need Steam Audio imho

[slate scarab]

not necessarily -- steam audio's HRTFs are 2-4x more performant than what we have currently, both with fyrox and sofar

[rapid hedge]

orly?

neat

Okay found the effect settings for Unreal, sec

FSteamAudioSpatializationSource::FSteamAudioSpatializationSource()
    : bBinaural(true)
    , Interpolation(EHRTFInterpolation::NEAREST)
    , bApplyPathing(false)
    , bApplyHRTFToPathing(false)
    , PathingMixLevel(1.0f)
    , bNormalizePathingEQ(false)
    , HRTF(nullptr)
    , PanningEffect(nullptr)
    , BinauralEffect(nullptr)
    , PathEffect(nullptr)
    , AmbisonicsDecodeEffect(nullptr)
    , PathingInputBuffer()
    , PathingBuffer()
    , SpatializedPathingBuffer()
    , OutBuffer()
    , PrevOrder(-1)

FSteamAudioReverbSource::FSteamAudioReverbSource()
    : bApplyReflections(false)
    , bApplyHRTFToReflections(false)
    , ReflectionsMixLevel(1.0f)
    , HRTF(nullptr)
    , ReflectionEffect(nullptr)
    , AmbisonicsDecodeEffect(nullptr)
    , InBuffer()
    , MonoBuffer()
    , IndirectBuffer()
    , OutBuffer()
    , PrevReflectionEffectType(IPL_REFLECTIONEFFECTTYPE_CONVOLUTION)
    , PrevDuration(0.0f)
    , PrevOrder(-1)
{}

[slate scarab]

potentially much more even, steam audio's spatialization probably scales way better for these high number of emitters

I can't test how it performs with this

RealTimeDuration(1.0f)
because it segfaults at the moment

audionimbus might be mixing up the sizes somewhere, causing steam audio to read past the end

[rapid hedge]

if you commit a branch I can show you the C++ code that segfaults

[slate scarab]

oh and to be clear, the direct effect still simulates occlusion

[rapid hedge]

Oh I thought you were talking about their default settings (which disable occlusion)

fair enough then!

[slate scarab]

As far as the audio processing is concerned, that's extremely cheap. Might not be cheap for the simulation though

im not keeping track of frame rates atm

[rapid hedge]

just did a test and can confirm we definitely don't lock for the reflection simulation 🙂

[slate scarab]

https://github.com/janhohenheim/bevy_steam_audio/tree/reflect-seg

feel free to check out

[rapid hedge]

on it

[slate scarab]

oh to be clear, the minimal example will tirgger it

oh the gltf example is cool

it's so sad when the playback ends and the sphere is despawned

[rapid hedge]

hehe feel free to set it to loop

[rapid hedge]

now let me investigate which variable it is

[slate scarab]

i guess the actual mistake will be kinda removed from the segfault
it must just be that the length of the IR buffer isn't correctly communicated somewhere

but who knows where that is

this is really making me appreciate memory safety by default 😅

presumably steam audio has a well written API, but it's basically just begging you to use it incorrectly (and produce a segfault)

[rapid hedge]

hmm can't find the right variable that's triggering this

but I'll send you the callstack

maybe that helps

chronologically:

and the relevant sizes, sec

[limpid mason]

So this is what "remote debugging" looks like in practice

[rapid hedge]

there Corvus, let me know if I should enter any variable in the evaluator

this / member vars work well, but any "regular" variables in functions are usually no longer accessible

[clear wasp]

Am I reading it right that it's segfaulting on absolute address 0x55? That feels like a null pointer + offset

[limpid mason]

A bit of topic, but did you have to do any setup to the debugger in rust rover, e.g. installing any additional tools?
I wanted to setup the debugger in vs code once but never got it to work

[rapid hedge]

Yeah you use this plugin: https://plugins.jetbrains.com/plugin/15090-pokemon-progress /j

jk, you just follow @oblique wren's excellent guide 🙂 https://github.com/TheBevyFlock/bevy_new_2d/blob/c0425fb185273cad60b6efff82a18271d7f944ed/docs/tooling.md?plain=1#L99-L113

It pretty much works out of the box

you just copy-paste the .idea folder in bevy_new_2d and set an env var

[limpid mason]

That does seem like an important part of the process

[rapid hedge]

I think you're onto something 👀

[rapid hedge]

maybe it's the use of [i][j] that gives that offset, but it's hard to know since j is optimized out

[limpid mason]

Thanks, I should probably try that out!

[rapid hedge]

oh also, smart pointers don't like the debugger

[clear wasp]

Can you set a breakpoint on one of the loops, step through it and see if any of the pointers are off?

[oblique wren]

I hope the guide helps you out! Unfortunately there's not really a good way atm to not require some manual setup of it, but at least it's documented somewhere.

[rapid hedge]

so I can't check if any of those contain NULL

[rapid hedge]

Don't believe so. This is using a precompiled .so file and the symbols file to check the source

I don't think there's a way to add a breakpoint there unless I compile it myself

[clear wasp]

I don't see why it shouldn't be possible. The source code should map onto addresses in the text section, and you can set a breakpoint on any one of those. But I haven't used lldb or rustrover, so maybe they can't

[rapid hedge]

fair

[rapid hedge]

All I know is that I can click to get the funny red circles on Rust files but not on the underlying C++ files

but what you say makes sense to me

[clear wasp]

Do you have any way of getting an assembly view for the current function?

[rapid hedge]

this?

[oblique wren]

Huh TIL there's a Parallel Stacks feature in Rust Rover for use with Multithreaded applications.

And I'm pretty sure there's a way to debug C/C++ files with RustRover, although you might need to use CLion with the Rust plugin to do it (but I don't know why you couldn't just use RustRover unless JetBrains decided that was how the segmentation was going to work...)

[clear wasp]

No, like godbolt for example, where it also shows the assembly code

[rapid hedge]

oh gotcha

uuuh

dunno

let me google

[rapid hedge]

there

[clear wasp]

Then you can maybe manually set a breakpoint: https://lldb.llvm.org/use/tutorial.html#setting-breakpoints

[rapid hedge]

ha, good idea

[rapid hedge]

yep that works

[rapid hedge]

Yeah definitely. I don't envy the task of setting up a safe Rust API around this

[clear wasp]

I made some progress on my Hdf5/SOFA lib today, I guesstimate I'm halfway done with the Hdf5 part. I got stuck for ages on a message that had a flags field, but also used the flags field of a previous header for a conditional field 🫠

[rapid hedge]

@slate scarab nvm we are definitely blocking lol

need to patch upstream audionimbus

there's a &mut self there that isn't actually true

well, it's "true-ish"

again, the tragedy of "you can do XYZ mutably on multiple threads, just don't call ABC"

@slate scarab

what do you mean no safety comment

ah

needs to be exactly // SAFETY:

lol

Anyhoot, I pushed my changes to no longer block the simulator

we now mostly just retry next frame

except in update_simulator where I expect the simulator to not be held up, so I return an Err if it's locked

Oh also I promised earlier to share this little nugget of wisdom

@slate scarab is there a strong need to split the encode and decode nodes?

Asking because it turns out the pathing stuff when set up properly will write 2 channels

[slate scarab]

yes

[rapid hedge]

So in unity what they do is directly decode the ambisonic channels into the 2 channel stereo output in the same node

and then on top mix the pathing stuff in

[slate scarab]

The advantage is that you only have to run the HRTF / decoding once for an entire group of spatial sounds.

What is the pathing anyway?

[rapid hedge]

uses baked probes

[slate scarab]

are you sure? https://valvesoftware.github.io/steam-audio/doc/capi/path-effect.html#_CPPv418iplPathEffectApply13IPLPathEffectP19IPLPathEffectParamsP14IPLAudioBufferP14IPLAudioBuffer this says it writes out to an ambisonic buffer

(Assuming we're talking about the path effect.)

[rapid hedge]

yep

[rapid hedge]

well see this here

by how this is written, I assume reflectionsSpatializedBuffer is stereo, right?

and here we can see that the pathing writes into that buffer as well, which is then mixed into the output buffer

which itself is the same size as the input buffer

so I assume the output buffer is also 2 channels

[slate scarab]

https://valvesoftware.github.io/steam-audio/doc/capi/path-effect.html#_CPPv4N21IPLPathEffectSettings10spatializeE

If you plan to immediately spatialize the output of the path effect
but we want to avoid that, since we don't plan to immediate spatialize

[rapid hedge]

Also, you will find this interesting. They have 4 processors, and I believe they are used like this:

• SpatializeEffect: does the whole per-source shebang, like our encoder node, but includes the decoding
• ReverbEffect adds scene-reverb on top
• MixerReturnEffect I don't get this one. It does something called iplReflectionMixerApply?
• AmbisonicDecoderEffect one-for-one our decoder, but I think this is for sources that already play ambisonic content: https://docs.unity3d.com/2022.3/Documentation//Manual/AudioDevelopAmbisonicDecoder.html

[rapid hedge]

hmm so you're saying we can just set that to false?

(that's how it is on main)

fair enough

in that case, pathing is already finished

[slate scarab]

Right, because the alternative would be to run the decoder for each node individually. My intuition is that this woud be bad, but I don't know how bad. It might be fine.

[rapid hedge]

then let's roll with your suggestion for now 🙂

I'm just confused why the pathing seemingly has no distance falloff

well it gets louder when you're near the source

but when you go away, it doesn't drop past a point

not even when totally occluded

[slate scarab]

Oh so the pathing is an alternative to reflections.

[rapid hedge]

I think they work in tandem?

[slate scarab]

https://valvesoftware.github.io/steam-audio/doc/capi/guide.html#pathing

Simulating reflections lets you model how sound propagates from a source to the listener. However, you may need to trace a very large number of rays...

For this reason, Steam Audio provides an alternative type of simulation: pathing.

With the trade off being you need to bake I suppose

[rapid hedge]

Doesn't it still use reflections as a fallback when setting it?

nvm

so it's either-or?

[rapid hedge]

there are also baked reflections FYI

separately from baked paths

[rapid hedge]

maybe that line is about baked reflections and just uses very bad words?

I'm skeptical because nothing in the plugins suggests that it's an either-or thing. There's no precendence of one over the other AFAICT

[slate scarab]

I mean they're separate effects; there's nothing requiring you to use both.

are you saying the unity integration for example applies both at once?

[rapid hedge]

I believe so

but take that with a grain of salt

I just mean that I see that both are turned off by default

and that the ui doesn't use radio buttons to suggest to use either-or

and that the processor will happily apply one after the other if both are set

which smells to me like they're complementary

Add to that that there's both

and

which makes me think that there's two kinds of baking: reflection and whatever path is

and again, these are all just best guesses from me

I may be very wrong

[slate scarab]

Hm, well we may want to do the same regardless. Since reflection is so expensive (and we can't break out the nodes yet), it might be good to be able to completely disable any of the three additional effects (pathing, reflection, reverb)

[rapid hedge]

Unity also uses either a binaural or panning effect on top

though that again is doing spatialization

btw do you know what directivity dipole weight and dipole power are?

audionimbus defaults both to 0.5, Unity defaults both to 0.0

[slate scarab]

i think that has to do with how "directional" a sound is, like its emission pattern
like how a loudspeaker pointed away from you isn't very loud

but i'm not familiar with the details

[rapid hedge]

what's your take on this?

[slate scarab]

It applies this on top of what?

[rapid hedge]

Another data point:

see how there is a check for realtime vs baked reflections, but it's not related to the pathing branch?

[rapid hedge]

right after the direct effect

and before all the others

[slate scarab]

ya seems like they could be complementary

[rapid hedge]

lol found a race condition in the unity code

let's uuuh not replicate that

[slate scarab]

It looks like there's a stereo path and an ambisonic path, where they eagerly decode all the ambisonic data to stereo for the former. The binaural effect is just kinda like our current HRTF -- it takes a mono input and produces a stereo output.

[slate scarab]

Well, actually right it's always a stereo output at most.

[rapid hedge]

I see 👀

oh hey Unity never calls the ambisonics_encode_effect equivalent

[slate scarab]

I think what I said actually incorrect 😅 they're very eagerly decoding everything

[rapid hedge]

I wonder if their pre-existing ambisonics pipeline takes care of that?

[slate scarab]

But they do branch often on whether they're applying binarual (aka HRTFs) effects.

[slate scarab]

Oh right, and the direct effect doesn't have anything to do with spatialization on its own -- it won't produce an ambisonic output. So the result of the direct effect has to be spatialized (assuming it's even simulated at all).

And since everything's eagerly decoded to stereo, they just write directly to the output.

[rapid hedge]

Oooooooooooh

yeah that makes sense

BTW before they apply the reflection stuff to the output, they ramp it between the old and the new values:

(same for pathing)

[slate scarab]

damn, so we need to do smoothing

[slate scarab]

I assume this is all done because it's easier to deal with in Unity if the output is stereo. It's either that or the efficiency losses from this eager decoding doesn't matter. I suppose it would be quite easy for us to profile.

[rapid hedge]

is that bad?

[slate scarab]

no it's just annoying and i was kinda hoping the effects managed that on their own

[rapid hedge]

Am I seeing this right that we don't need that binaural effect?

Since we already do spatialization always?

[slate scarab]

ya we don't need it (and we don't use it!)
we spatialize the final mixed ambisonics in the decoder node

[rapid hedge]

got it 🙂

Also, fun fact: the output params the simulation hands out are not necessarily complete. Idk if this is really documented anywhere, but apparently you're supposed to fill some fields out on your own and leave some others at the values the simulator hands out

[slate scarab]

btw idk how this escaped me, but we absolutely don't need to calculate the reverb individually for each emitter

[rapid hedge]

yep 😄

Mind PRing that?

I think you can refactor that cleaner than I can

[rapid hedge]

^ same as unity in that case

[slate scarab]

ya that's what the mix return is (aka a send!)

[rapid hedge]

Oooooh excited to see that in action!!

[slate scarab]

it also means we don't need the proc store for the moment, which is kinda lame

i like it

[rapid hedge]

me too!

Hmm I think we have almost everything from the Unity plugin covered then

[slate scarab]

unless we wanna put the steam audio context in there

[rapid hedge]

I prefer the static one tbh

[rapid hedge]

What's missing is (apart from user configurability):

• That audio ramp
• whatever the heck the global reflection mixer is

[slate scarab]

ya that's the reverb

[slate scarab]

this guy

https://github.com/ValveSoftware/steam-audio/blob/master/unity/src/native/mix_return_effect.cpp

[rapid hedge]

Oh hey you're right!

I meant the MixerReturnEffect

[slate scarab]

oh

[rapid hedge]

that is the one I don't get

lemme show you the processor

lol discord converted my ```csharp markdown annotation into a ".csharp" file

Don't tell me syntax highlighting works on this

phahaha

Wonder if a C# compiler would agree with that file ending haha

some kinda ambisonics decoder?

Which I find weird, given that they also have a dedicated AmbisonicsDecodeEffect:

I don't really get why there are two different decoders

Oh, also, Unity has a thingy called "Perspective Correction", to which I found this:

seems worthwhile?

[slate scarab]

No it looks like a performance optimization for reflections. Instead of just applying like we do now, we'd call https://docs.rs/audionimbus/latest/audionimbus/effect/reflection/struct.ReflectionEffect.html#method.apply_into_mixer

(that would be a good proc store use)

idk why it's better

[rapid hedge]

ooooooh I see

cool!

perf optimization for reflections sounds good

[slate scarab]

so that in combination with one reverb calculation should speed it up a decent bit

[rapid hedge]

heck yeah

that's precisely our bottleneck

I just wish I knew what the heck pathing was

I begin to suspect that I'm not supposed to just draw an AABB over the whole scene for that

since the pathing is global that way?

But then again it does say in the docs "something something only over this threshold are they mutually visible"

which sounds like it shouldn't go through walls

oh hey when I change the mesh to static it doesn't generate any probes

I don't get this

Unity also uses a static mesh

[slate scarab]

The docs describe pathing as a way to calculate the shortest, non-occluded path between a source and the listener. Both the reflection calculations and the pathing are intended to simulate this aspect. But I thnk reflections explore more possibilies, so it's not like pathing is just a faster reflection simulation.

[rapid hedge]

that makes sense to me

[slate scarab]

It seems to be a more reliable subset of reflection, basically.

[rapid hedge]

okay, I know now why it didn't generate anything for static meshes

~ race condition ~

still, the pathing is behaving like it's never occluded

[rapid hedge]

Hold up, is our HRTF even working?

try this

the horizontal panning clearly works, but I don't hear anything for vertical movement

[potent moat]

should be invited now @rapid hedge

[rapid hedge]

thx!!

[rapid hedge]

@slate scarab alright, here's a PR for eager spatialization. Check the new spatial example 🙂
https://github.com/janhohenheim/bevy_steam_audio/pull/3

IMO this version sounds much richer

I removed the reverb in this PR in anticipation of your reverb node PR

[rapid hedge]

Using this branch and tweaking my settings a down + using 1st order, I can comfortably simulate 250 probes using both direct and reflection 🙂

[slate scarab]

Hm, I'll have to come back to this in a bit. Although I'm a little worried -- I really don't see how running an HRTF for every emitter could be better than one for all emitters.

Technically it's two right now for the reflections, but if we use the mixer then they'd be funneled into one output anyway.

[slate scarab]

Ah, right, it's actually a trade off. Direct HRTF spatialization won't lose any fidelity from the ambisonics order. But I think I'm correct about the cost; ambisonics encoding (with one decoder) scales much better than an HRTF for each source.

Ideally, then, you'd be able to mix and match. Important or nearby sounds should get a dedicated HRTF, whereas less important or far away sounds would do fine with ambisonics, even quite a low order.

Either way, I don't think we should just delete the decoder.

A really neat benefit of the ambisonics encoding is that you're not stuck with stereo output, too. If you're doing an installation or you just need a different speaker layout like quad or 5.1, then the sound field allows you to easily translate to arbitrary layouots.

All that to say -- I think we can pretty easily write an eager and lazy spatializer node. Both of these should probably use the reflection mixer, so the only real difference will be using a binaural effect and spatializing the pathing immediately vs using an ambisonics encode effect and not spatializing the pathing.

that's my opinion anyway, lmk how you feel about it

[tender fiber]

I did 7th order ambisonics in Reaper. So many tracks it boggles the mind.

And It was cool

I did 7 channels and downmixed at the final stage so 1 fn instead of many is definitely the path forward

Hence the track count increasing by a lot x7

And all required their own positioning fn in space

to get 7.1 (minus .1)

In Ambisonics

Stereo (my computer hated me for a week)

[slate scarab]

Anyway, I think I'm satisfied with the ProcStore API for the moment, and I've integrated the playback changes in bevy_seedling. @dusky mirage, do you feel comfortable merging freeverb and the pre process nodes for a final round of testing? I feel like it's at a good place for another release, and it would be nice to publish a proper release for Bevy 0.17.

(It'll be a touch easier to test the pre process nodes in practice if everything's on main.)

[tender fiber]

I got Firewheel CLAP node to host using clack Billy. Talking to Robb’s plugins is another task all by itself and I wanted to split it up. Thoughts Billy?

[tender fiber]

… Besides Hello Meadowlark

[rapid hedge]

I think I agree in principle, but I would bench it before adding complexity

Which should be easy since we have both approaches already in code

[rapid hedge]

I love the fixed block thingy you added! Do you plan on upstreaming that too?

[slate scarab]

ya i could make a PR there's still the question of guaranteeing no dropped samples, which is kind of annoying

rather than how it is now, which should work well in practice, but could, given certain settings and circumstances, produce stutters

so i'll think about that

[dusky mirage]

Ok, sure!

merged!

[dusky mirage]

Oh sweet! I'd like to see it!

[rapid hedge]

@cold isle mind taking a look at my followup PR? 🙂

[rapid hedge]

@dusky mirage is there a simple way to set up a processor for benching in criterion?

I'm trying to create it manually, but there's a lot of structs that need to be initialized in order for .process(...) to work

I also did a bench for the whole Bevy app, but since Firewheel will drop audio events when it starts lagging, the result of the benchmark is rather uninformative

Can I tell firewheel to plz start blocking the main thread if it takes too long?

that way I would have factored in the time for the processor to run

[slate scarab]

You could look at the testing backend I have for seedling to get an idea of how you might make an end-to-end profiling backend. https://github.com/CorvusPrudens/bevy_seedling/blob/master/src/utils/profiling.rs

(ignore the name it's not actually for profiling)

You'll need a static or something to actually get at the processor, which is kind of annoying but very doable. The testing backend spawns a thread.

But rather than spawning a thread, if you write it into a static, then you could pull the processor out of the static and then do profiling with it.

[rapid hedge]

thx!

I assume the usage is here? https://github.com/CorvusPrudens/bevy_seedling/blob/5e11f13253754dd837a926db49f0cd33da2ca689/benches/basic.rs#L2

how come it's commented out

[slate scarab]

no i use it for testing

[rapid hedge]

Oh it's not pub

[slate scarab]

ya it won't work as-is

for profiling

[rapid hedge]

heck

[slate scarab]

It just demonstrates how to write an extremely minimal backend

i can write a profiling backend maybe later today 👀 we really need one anyway

[rapid hedge]

that would be neat!

Then we can check how much more expensive the eager eval is 🙂

Is it okay if I merge the eager version in for now?

And then add the decoder version if it's too expensive?

[slate scarab]

If you do I think I'd PR in a decoder and lazy encoder either way. I really want to preserve that functionality!

If you don't mind of course.

[rapid hedge]

Sounds good to me!

merging now then

[slate scarab]

I'll have to double check, but I don't think there's any real reason for the new spatializer node to have stereo inputs. If that's correct, we can bring it back down to one input.

[rapid hedge]

I followed Unity and Unreal's example. They both make a difference between "use mono input for these effects" and "use stereo input for these other effects"

I don't know if it's necessary or not

I also added the audio smoothing btw

Adding some issues for all things in my mind

[slate scarab]

The docs mention that the direct effect can provide partial occlusion. It's possible that as well as the directionality parameters interact with a stereo stream in a meaningful way.

[rapid hedge]

You do have to explicitly set the amount of input channels for the direct effect

while others are just hardcoded to mono

so I would assume there's a reason, yeah

There, dumped some: https://github.com/janhohenheim/bevy_steam_audio/issues

I think you said you could do the reverb mixer, right?

Thx 🙂

[slate scarab]

it seems kinda trivial to be honest

we can have it be conditional on whether a mixer is present in the proc store

which is neat

oh that's for reflections

[rapid hedge]

I think once we have that I'm comfortable with publishing a 0.1.0 version for people to try out!

[slate scarab]

The reverb might actually be annoying because of the channel layout

[tender fiber]

@dusky mirage PR is up!

[slate scarab]

actually I think it's fine

[tender fiber]

Just don’t make fun of my Cargo.toml file or I’ll personally call Tom.

It is, after all, Tom’s File

[rapid hedge]

Just to make sure, you're talking about https://valvesoftware.github.io/steam-audio/doc/capi/reflections-effect.html#_CPPv418IPLReflectionMixer, right?

[slate scarab]

Yes, that's the performance optimization for reflections that use convolution (which is the current default).

[rapid hedge]

Here's the relevant Unity code for reference:

btw you need to check out the spatial.rs example I added 😄

it's a bit of a misnomer, it's just showing direct hrtf stuff

but it feels nice

[slate scarab]

it does mean magical connections that aren't expressed in the firewheel graph, but... reflections are too expensive 😅 imo definitely worth the indirection

[tender fiber]

Please be easy.

[rapid hedge]

I can also write a new reverb node once the mixer is up 🙂

[dusky mirage]

Cool! One thing I would like is if it made use of the the constant mask feature of CLAP. That way external plugins can interact with Firewheel's silence mask optimizations.

[tender fiber]

I only got it registered using host and read by custom_node example. CLAP params write changes are not impl

Yet

No mask in code

It had to read ID and location and match both in order to read the plugin and add to custom_node example

So no CLAP (2/2) only clack host (PR) (1/2)

2/2 is reading enum Params from robbert’s CLAP plugin code (with your mask optimization) to write changes to plugin

Also, you had egui hopes so that might be a real thing as well.

(1/2) is all manual. I tried to write error messages so others would PluginNotFound struggle like I did.

Code was getting up there so I wanted to split that.

And you might have a Firewheel folder cargo.toml that’s much better. I just wanted to compile. I hope you can fix my mistakes. Sorry about butchering that.

[tender fiber]

@dusky mirage code won’t work if you run from outside nodes directory. It will error running from Firewheel folder.

That manual location thing…

[cold isle]

Would you mind listing the segfaults you’ve encountered here so that I can fix them when I get the time? https://github.com/MaxenceMaire/audionimbus/issues/29
This really shouldn't happen since AudioNimbus is meant to be a safe wrapper

[slate scarab]

One I've encountered seems to just be running the reflection effect with an IR time of less than two seconds. I should probably gather more detail on that, though.

jan dumped the callstack earlier for it

[rapid hedge]

damn I should have kept notes

hard to remember what the causes were after the fact :/

my bad

[cold isle]

That's already helpful, thanks for sharing! I suppose Steam Audio has a list of possible errors somewhere since those are super easy to come across

[rapid hedge]

There's also the matter of thread safety

[cold isle]

No problem, I'll try and reproduce them 🙂

[rapid hedge]

Not sure how to best express these invariants, but basically, set_inputs and get_outputs are threadsafe as long as no simulation is running for the specific flags you passed

And almost everything else is threadsafe until you want to run .commit() on the simulator

well, "thread safe"

you cannot add a source from two threads at the same time, BUT you can add a source while a simulation is running, just don't commit

With the exception of adding a probe batch, that is never allowed to run parallel to a simulation

[cold isle]

ahh the joys of FFI 😩

[rapid hedge]

Yeah this really doesn't map well onto Rust

[rapid hedge]

I changed some &mut selfs to &self so that the firewheel / Bevy integration can fully utilize that parallelism model

at the cost of the API being a bit less safe as a result

[cold isle]

it's tricky with Rust's type system only but it should be doable with runtime checks

[rapid hedge]

Also, a lot of FFI calls store shared pointers in the end. There's quite a few places in audionimbus where you pass &some_handle, but actually, the handle gets cloned on the C++ side

So I started changing those to take the handle owned

to make the user call .clone() on it to know "aha, I'm passing a handle to this thing and now it owns it!"

[rapid hedge]

There's also shCoeffs which are some random pointers to values that are only meant to be looked at juuuust before applying the effect

but don't you store them anywhere, or else

I really hope I make it clear that any shortcomings in audionimbus' safety invariants are definitely not your fault

I have no clue how I would design a safe wrapper around this 😄

[cold isle]

Totally, in fact I'm very happy that you guys are actively using it and taking the time to improve it 🙂 To be honest I haven't had a chance to use it myself thoroughly yet, so you're paving the way and I'm thankful that you raise the issues instead of ignoring it and moving on to something else

[rapid hedge]

If you have time, try the examples 😄

I think you'll like them!

[cold isle]

I hope to free up more time soon, it’s a bit of a rush right now. Your plugin looks very promising, though, so I’ll be happy to explore it for my own projects 🙂 We should also be able to merge the changes to the demo soon, once the AN changes are out of the way

[rapid hedge]

I addressed your feedback on https://github.com/MaxenceMaire/audionimbus/pull/27

[cold isle]

Cool, thank you!
If you're happy with the changes I can release the new version right now

[rapid hedge]

Yes please

@dusky mirage hrmmm changing from firewheel = { git = "https://github.com/corvusprudens/firewheel", branch = "freeverb" } to firewheel = { git = "https://github.com/BillyDM/firewheel" } in bevy_steam_audio gives me this

Looks like these are the commits that changed between those states:

[slate scarab]

we are still adding a zero input zero output node, right?

[rapid hedge]

yep

[slate scarab]

i wonder if that's triggering it

[rapid hedge]

yep that was it

But the audio is still silent

Oh I bet I query for it

sec

[slate scarab]

or it injects state into the proc store that the nodes want

[rapid hedge]

it was a Single 🙂

[rapid hedge]

but yeah this is from the cool 0->0 node

[slate scarab]

we observe the interesting phenomenon where anything you don't test will necessarily fail

like a murphy subset

[rapid hedge]

As a wise crow once said

@cold isle yay, it runs 😄

No more fork needed, thanks!

[cold isle]

Great 🙂 just waiting for the crypto bros to finish generating their docs to see if everything's good hah

https://docs.rs/releases/queue

[rapid hedge]

pfff hahaha

oh hey a bevy

AND a symphonia

[slate scarab]

now if we could just get symphonia to publish a relase... 😅

[rapid hedge]

I'm this close to forking it to patch those damn logs out downstream

[cold isle]

Didn't they release a new version a few days ago?

https://github.com/pdeljanov/Symphonia/releases/tag/v0.5.5

[rapid hedge]

Oh hey!

Lowered the log level of some informational logs to be less annoying.

[cold isle]

They heard you 😄

[rapid hedge]

Indeed there's fewer logs!!

[rapid hedge]

apparently not clear enough

ah no nvm the logs that are left are ALSA

heck yeah

[slate scarab]

this day brings good tidings

[rapid hedge]

I'm so happy to be rid of those gigantic symphonia logs

this is awesome

[slate scarab]

well that should help give people a much better first impression of bevy_seedling 😅

[dusky mirage]

👀 https://github.com/RustAudio/cpal/blob/master/CHANGELOG.md

[rapid hedge]

@slate scarab I went ahead and reimplemented a dedicated singleton reverb node. We can then plug your reflection mixer in it when it's ready 🙂
As with the regular node, it's pretty much a 1:1 port of unity. Mind checking it out?

https://github.com/janhohenheim/bevy_steam_audio/pull/12

[rapid hedge]

Oh, also added the decoder node back in

[dusky mirage]

Hmm, shoot, I just realized that we wouldn't be able to publish Firewheel if we used clack as is, since crates.io doesn't let you have dependencies from git. We'll need to ask the developer to publish clack on crates.io.

[tender fiber]

I hope they will. Mask surface feature impl just fixing up stuff now. Firewheel is talking to plugin now.

[limber kernel]

looks like he started a thread in the rust audio server!

oh youre already there :)

[tender fiber]

yes. thx

With CLAP
Better effects
Even doomy’s

So hopefully, they’ll say yes.

[rapid hedge]

Any idea why I'm still getting this wall on Foxtrot?

This is the right version

[slate scarab]

well i've got some bad news

[rapid hedge]

:<

[slate scarab]

they only reduced the level of some of their almost useless logging

[rapid hedge]

Wowee

Welp, this works:

DefaultPlugins
  .set(LogPlugin {
    filter: format!(
        "{default},symphonia_format_ogg::demuxer=off",
        default = bevy::log::DEFAULT_FILTER
    ),
    ..default()
})

[slate scarab]

very frustrating though

They have their reasons, of course, but boy is it bad DX for almost all downstream users.

[rapid hedge]

certainly their reasons could be handled by, say, trace or debug

BTW for the steam audio stuff I think I'm happy with the core feature sets as-is 🙂

[slate scarab]

While I'm complaining, I really don't like the reasoning, here.

[basically just filter it out]

That's likely the best way forward since the log facade is very popular, and it's pretty unlikely that all your dependencies will only ever emit error or warn messages.

(look who made the issue haha)

[rapid hedge]

hehehehe

[rapid hedge]

That's such a false dichotomy holy shit

[slate scarab]

My... uncharitable paraphrasing would be "other crates might not be well behaved so we won't bother." However, the reality is that all like 400 default Bevy dependencies are more well-behaved than symphonia, who is the outlier here.

[rapid hedge]

Yeah I don't mind info logs

this is a great info log!

the issue never was "info logs in general"

[slate scarab]

Anyway, I don't mean to beat a drum or anything. Symphonia is a pretty foundational crate, and a lot of hard, often thankless work has gone into it! I just really don't like how I can't really do anything about it and it makes my crate a worse experience than it should be.

[celest whale]

I left a complaint wearing my Bevy maintainer hat >.>

[rapid hedge]

Should we add symphonia to our default filters when we upstream seedling?

[celest whale]

Yes

This is unacceptably bad

[rapid hedge]

Same for that calloops thingy

Not sure you remember

[slate scarab]

oh i didn't realize there were default filters

[rapid hedge]

@slate scarab ^

[celest whale]

Actually, can you add it now? We have optional symphonia features already using bevy_audio

[rapid hedge]

Sure

[slate scarab]

-# oh i did the thing where i just kinda looked at it and went "ya that seems good"

[slate scarab]

So maybe this is a little overstated. (Actually only naga and wgpu are in the default filters, so... I think my point still stands.)

I wish there was a good way to add to the default filters as a third party crate I could do it if the filters were a resource.

[rapid hedge]

I honestly also read it that way

[rapid hedge]

That should be entirely doable

We could move the log stuff to PreStartup

[slate scarab]

now we're talking

[rapid hedge]

Then any third party crate has the opportunity to inspect the default log levels (in a resource)

https://github.com/bevyengine/bevy/pull/21548

There ya go @celest whale

[slate scarab]

oh no it makes typos mad

[rapid hedge]

how do I tell typos to shut up?

[celest whale]

typos.toml

[slate scarab]

-# maybe we ought to tell symphonia it should really be "calf"

[rapid hedge]

But I don't necessarily want to add that there

Is there nothing like #[rustfmt::skip]?

full logs hehe

@slate scarab review plz https://github.com/bevyengine/bevy/pull/21548

Furthermore: review plz https://github.com/janhohenheim/bevy_steam_audio/pull/12

[slate scarab]

wow mr gallups beat me to it

[celest whale]

Merging

[rapid hedge]

heck ye

[dusky mirage]

Don't forget to add filters for all the other symphonia decoders/demuxers!

[rapid hedge]

I added one for each format

[rapid hedge]

@slate scarab I changed the reverb node to just take mono input, but that gives me

2025-10-15T02:46:21.530147Z ERROR bevy_seedling::edge::connect: failed to connect audio node to target: Input port idx 1 is out of range on node NodeID(Index { slot: 56, generation: Generation(1) }) with ChannelCount(1) input ports

I'm a bit confused, I thought seedling now automatically downmixes stuff?

Hmm, I suppose that's because it's not a sample effect

I suppose I need to add some kind of downmix node before the SendNode then?

Is there a builtin node for that?

I'm also merging the reverb stuff in because I'm about to write an audio scene backend, and that requires a crate reorganization 🙂

please correct me if my understanding of pools is wrong in this case!

[drowsy dome]

im gonna miss those logs 😢

they were really informational

[slate scarab]

This is something that I should fix in bevy_seedling.

[slate scarab]

Hm, this doesn’t make total sense to me. You’d still want your footsteps to engage with reverb in certain spaces, for example.

[slate scarab]

The spatial node does not include reverb, so it's not really the full effect (if you consider the distinct effect of reverb necessary for the full effect, which I think is fair).

[rapid hedge]

Fair enough, you’re right

How should I wire it up then?

SendNode to reverb node?

[slate scarab]

Well, what you did earlier should "just work." But following the speaker-based mixing changes, the SendNode needs some additional handling that I didn't add.

For now, don't worry about leaving it as-is. I can PR in a send node after fixing it in bevy_seedling.

[rapid hedge]

Alright 🙂

[rapid hedge]

And then you'll add it to the SA pool, right?

[slate scarab]

Sorry about my glacial contributions by the way 😅 this work is very exciting!

[rapid hedge]

Wonder how the user sets the gain of the reverb

All other gains are straight on the node

[slate scarab]

It is a touch ugly since they'd need to specify the steam audio reverb bus, but we could wrap that up in a function.

[rapid hedge]

You’re doing precisely those things I cannot do, so I'm really grateful

[slate scarab]

That is, right now it would be

commands.spawn((
    SteamAudioPool,
    SamplePlayer::new("stuff"),
    sample_effects![
        SendNode::new(Volume::SILENT, ReverbBus),
    ],
));

But we could just make a function that returns the right send node

commands.spawn((
    SteamAudioPool,
    SamplePlayer::new("stuff"),
    sample_effects![
        reverb_send(Volume::SILENT),
    ],
));

Updating it after the fact is a bit simpler since you can just mutate the SendNode, so this mainly a spawning ergonomics question in my opinion.

[rapid hedge]

Sound good!

The only think that nags me a bit is that all of this assumes a lot of knowledge about seedling

And I don’t think the onboarding is currently strong enough to give that to audio newbies like me

But that’s not an SA problem

It’s fine if SA just assumes you know your stuff 🙂

[slate scarab]

Would you prefer if the reverb volume were expressed on the SteamAudioNode, or even an additional component on it? After all, we could trivially write a system that reads that and writes it to the send node if it exists.

[rapid hedge]

Thing is, now that I know seedling better, the API above using a send is much cleaner

I wonder how this will look post upstream

Will we prefer things to be more component shaped? Or more node shaped?

[rapid hedge]

It’s hard to tell. On one hand, this is much more ergonomic. Or the other, it makes the graph more magical, and there is right now an extremely low mental overhead to learning what the SA pool does (assuming seedling knowledge)

You basically take a single glance at the definition and are like "yep that’s how the data flows"

(With the exception of SteamAudioSamplePlayer which I intend to kill)

[slate scarab]

Hm, then in some sense do you feel seedling's API might be bordering on too low-level for new users?

[rapid hedge]

Honestly not really

Once I had my private tutor crow on Discord it all clicked

I mainly think that we could retarget the docs a little bit to consider newbies, and maybe have a little "getting started" guide like Bevy that guides you through building your own simple effects

Or something like that

But the end-user API is lovely

[slate scarab]

There's certainly some irreducible complexity somewhere in the system. My belief is that the graph API is elegant, powerful, yet not too difficult to get started. But that last part might be a little hopeful 😅

At the same time, I think these questions basically go away with a node graph editor.

[rapid hedge]

Yeah definitely lol

The graph API has just enough magic in it that the onboarding is a liiiittle bit steep, at least for me

Like the order in which sampler effects vs nodes directly on the sampler are evaluated

The fact that there are 3 nodes that get spawned using pools and that only one is „real“

Then there’s a lot of terminology that I didn’t get

Like "send"

That AudioEvents are something entirely different than Bevy events or messages

But the API itself once you learn it is really really really good

Like, extremely good

[slate scarab]

This one's tough since we really want the sample_effects relationship to serve two roles; as a template for the effects of a pool (which are then inserted into the graph when a new sampler is added) and as a way to set their values when spawning a sample player.

[rapid hedge]

So all this to say: no the seedling API is not too low level in my opinion 🙂

[rapid hedge]

Alice, if you’re reading this, this is not an upstreaming blocker. The stuff that I found hard to get into are things that seem literally impossible with current bevy_audio

[slate scarab]

I think we could probably eliminate the second one by applying these values to the real nodes and then hooking up the SampleEffects relationship to the real ones (despawning the "temporary" ones). But that is also a little magic.

[rapid hedge]

Yeah I don’t know the best way forward either. Maybe there’s some insight from Cart with BSN

This is inherently a bit tricky!

Maybe indicative that there’s just no elegant way to initialize related types in Bevy during spawn time

[slate scarab]

Well I think the trick is more that we want to set the values of a set of components that we know exist somewhere, but which we don't yet have access to. The SamplePlayerhas to wait for a slot in the pool, after all.

[rapid hedge]

Yeah that is better put, thanks

[slate scarab]

If we did this switcheroo, there would be a moment at which those related nodes go from being placeholders to being the real deal, which.... maybe also isn't good.

[rapid hedge]

If this was #ecs-dev, this would be the moment when Sander swoops in like "flecs has an API for that" and shows a snippet of the most beautiful abstraction on earth

[slate scarab]

actually that is probably true

You might be able to express the nodes with IsA, which maybe makes that relationship a bit clearer

[rapid hedge]

BTW I want to implement Avian support today. I have some very specific ideas how to do it, so this should not be too hard. One thing that I was mulling over is whether to require annotating colliders or not

Currently for the mesh backend, all meshes that are valid steam audio geometry must have a SteamAudioMesh

that mesh tells me

• is this a dynamic or static object
• which accoustic material does it have

I want to split that into two components, since static-ness is kind of a micro-optimization anyways and we can infer that from avian

so that's easy

But for the second one I'm not sure

I'm currently tending to "the user must annotate colliders (not rigid bodies) manually with the material for SA to pick them up"

Which is just easy enough to forget that I might want to add a DebugPlugin with gizmos to show all SA meshes

[slate scarab]

oh that would be awesome

[rapid hedge]

can even have different gizmo colors for different materials

We could make all colliders implicit SA meshes

but that requires some bookkeeping (e.g. Sensors should not be SA materials, until you remove the Sensor component, same for colliders without rigid bodies)

[slate scarab]

Right, but that might destroy the simulator

[rapid hedge]

should be fine perf-wise

It's all simple shapes in a BVH

it's more about that there are a lot of edge cases to be handled automatically

and we should have escape hatches for all of them

For rerecast, this was a bit simpler to handle, as it only needs to be in a correct state when the navmesh is generated. So that system just needs to have the right queries in place.

Thus, in Rerecast you never need to annotate anything. Avian Just Works.

But in SA we are using lifecycle observers to continuously keep Bevy and SA in sync

[slate scarab]

I suppose you'd want to be able to configure the default material in an automatic scenario

[rapid hedge]

There's Material::GENERIC

but yeah setting that to e.g. wood would be neat

[slate scarab]

if you're in a stone building you might want stone / brick

etc

[rapid hedge]

So I think an API where everything magically works for Avian is possible

[slate scarab]

Maybe it should even propagate along relationships 😅

[rapid hedge]

can you elaborate?

[slate scarab]

If a building is expressed as a hierarchy of entities, and it's like all stone, it would be neat if you could mark the top-level entity as having a stone material.

[rapid hedge]

Oooh right

[slate scarab]

idk if that's all that common a representation

[rapid hedge]

@sturdy prawn I think nothing works like that in Avian rn, right?

[slate scarab]

but it could take a lot of manual work out maybe

[rapid hedge]

Like, I think I can't set a rigid body to a specific collision layer and have all colliders inherit that, right?

Or set the default collider density on a rigid body

(without a collider constructor hierarchy ofc)

If my memory is correct in that this does not work, I would also not support it in SA

even though it's neat

[deft jasper]

oof that is a rough acronym to use 😅

[slate scarab]

hello I'd like to upstream SA

[rapid hedge]

oof I didn't even notice

you're right

BSA it is

-# pronounced B-scena

[sturdy prawn]

Correct, though idk if you could use Bevy's new component propagation thingy for it, I haven't checked how it works

Do y'all happen to have a use case for non-Parry convex hull generation for audio meshes or something else btw? I'm cleaning up my Quickhull implementation right now

(and implementing the 2D version)

[rapid hedge]

In the end we need trimeshes, so I don't think so

I do need upstream trimesh everything though 😛

What do you think about this API?

let mesh = collider.trimesh_builder().build().unwrap();
let mesh = collider.trimesh_builder().sphere_subdivs(16).build().unwrap();
let mesh = collider.trimesh_builder().capsule_subdivs(5, 10).fallback_subdivs(30).build().unwrap();

[sturdy prawn]

That's pretty neat

[rapid hedge]

can I upstream plz 👉 👈

(and get it in 0.4.1)

otherwise I need to make a microcrate for this

since I now need it across 3 projects

[sturdy prawn]

Sure, I'll allow it :P

It shouldn't be breaking so shipping it in 0.4.1 sounds fine

[rapid hedge]

Hmmm I thought about this some more

Reverb is just reflections where listener = source

So for your own footsteps, wouldn’t you want to just use reverb?

And for other NPCs, wouldn’t you use just spatialized sounds?

[slate scarab]

In practice maybe, since it's a little disorienting to have very nearby sounds (like your own footsteps) spatialized. But in theory, no.

[slate scarab]

No, because reverb is a more emergent phenomenon than mere reflections.

They do different things.

Even if you can express reverb with the reflect effect.

[rapid hedge]

How is it different when in Steam Audio they use the exact same method?

[rapid hedge]

I suppose reverb vs reflections is a bit like ambients light vs global illumination?

[slate scarab]

That the reflect effect can express reverb if you invert the relationship does not make it mutually exclusive with other spatialization methods.

As an example, a delay effect (which you can think of as echoes, really) is very distinct when the time between echoes is very long. But when you reduce the delay time to a handful of milliseconds, it starts to sound like a short reverb. In a sense, it is a short reverb. After all, reverb isn't much more than a jumbled mess of reflections that are so mixed up you can't distinguish individual reflections anymore.

[tender fiber]

Delay is not changing the room. Reverb is.

[slate scarab]

In reality, reverb tends to mess with phase relationships in a somewhat more complex way, but I'm not surprised steam audio's reflect effect can capture that.

[tender fiber]

Ideally, you want a low pass filter after your reverb to stop cave rumble of your stage performance

high pass Sorry

Like a lot
So 1 khz-4khz gets through

LP and HP are crucial to clamp unwanted reflections because reverb doesn’t care

That whole phase thing I was saying.

[slate scarab]

I don't have any deeper insight than that, though. I'm not exactly sure what is even being simulated. My main point is that it's not inherently surprising that the same effect can be used to model distinct phenomena.

[oak walrus]

Well, everything is delays in the end

Filters are delays, choruses are delays, phasers, flangers, reverbs

[tender fiber]

@dusky mirage Someone got a a crates.io clack placeholder crate on crates.io. Can we just fork clack?

Anything analog IZ delay

[oak walrus]

Digital too

[tender fiber]

delay compensation
jk

[dusky mirage]

Possibly. I'll wait a bit longer to see if the original author responds.

Ah yeah, it's common practice to put a high pass filter on a reverb. It should be pretty simple to add that to the freeverb node itself (just apply it to the wet signal).

[dusky mirage]

Or users can always just add filter nodes after the reverb node (though that would only work if the wet/dry mix is at 100%).

[tender fiber]

MIX definitely

Otherwise, mix is just reverb

[rapid hedge]

@slate scarab we have Avian support now 🙂

It works automatically. Just add AvianSteamAudioScenePlugin and every collider that is eligible (has a ribid body, is not a collider), is turned into a steam audio mesh and updated at runtime!

Next step: TrenchBroom integration

@safe grove what did we say? Use the entity BTB creates per TB material to figure out the material?

Could even just use that render mesh as the steam audio mesh instead of the collider. They should be identical, right?

(need a trimesh in the end anyways)

[safe grove]

Yep! (Probably through a property in the GenericMaterial)

[safe grove]

Sounds right to me

[rapid hedge]

I thought it could also be neat to have some way of saying "all textures that have 'wood' in their name are wood"

Though maybe using inherited materials would be better for that

[rapid hedge]

When using extra properties, the bevy object still holds a MeshMaterial3d in addition to the generic material, right?

[safe grove]

Yes

[safe grove]

I honestly don't remember, it's been a while since I've touched bevy_materialize

[safe grove]

Probably not actually
If you want to not have a material metadata file, then yeah, name tagging would probably be your best bet
If you do, then most likely you'd have to have a pbr_wood.toml, pbr_rock.toml, etc. that you'd inherit from, would be much easier just to use [properties]

[rapid hedge]

I guess I could take in a HashMap connecting regexes to Steam Audio materials

Could maybe even have that for regular MeshMaterial3ds too, possibly

I believe we can look up their asset path

[rapid hedge]

I'm still figuring this whole OBS thing out, so don't mind the smol window

but this is Steam Audio + TrenchBroom working 🎉

this is the setup in TB

And then when you load the scene, bevy_steam_audio automatically generates audio materials for everything, based on the name of the material 🙂

@safe grove

[rapid hedge]

It might be a bit subtle, but notice how when I enter the moss room, the whole audio becomes more muffled!

@slate scarab I think this is the last big feature I personally needed from steam audio

I'll write some docs for it now, but other than that I'm probably not touching it for now, at least until I need baked reflections

or someone explains to me how the heck pathing is supposed to work

Oh also when you do your reverb and mixer PRs, could you add yourself to the crate authors in the workspace? 🐦‍⬛

@cold isle here's it all working with dynamic physics stuff btw. I remember you were asking about the transform propagation 🙂
It sounds a tiny bit off because the audio source is in the center of the ball and because sometimes cubes go into the player

[slate scarab]

now you just gotta use it for AI

[past sandal]

Did you set the occlusion type to volumetric and give it some samples + a radius? I think that can help with the abrupt changes in the audio 🙂

[rapid hedge]

Nope, it's raycast!

Thanks for the advice, will try when I get back to it 🙂

Bookmark to self: GitGhillie occlusion raycast volumetric abrupt

[rapid hedge]

@cold isle got a tiiiiiny new PR. Mind releasing a patch? https://github.com/MaxenceMaire/audionimbus/pull/30

[rapid hedge]

I'm running into the same segfault now :/

Did you figure out the fix?

[slate scarab]

I think you can specify the size in multiple places (init and usage). The idea being any size smaller than what it was initialized with can save on performance. But apparently it just segfaults in the other case.

[rapid hedge]

I'm using our default settings

so I would have hoped we use the same duration everywhere

You're talking about this guy, right?

HA! Interesting. When I set the sound to loop it does not crash

it's a very very shot footstep sound

oh nvm it still crashes, it just takes way longer to crash

[slate scarab]

oh well i never ran into it with default settings

[rapid hedge]

Made a minimal repro: https://github.com/janhohenheim/bevy_steam_audio/pull/27

interestingly the audio also becomes a loud "BUMP" sometimes

Do you think you could check it out if you have time?

I'm really puzzled

[slate scarab]

ya i should have the opportunity this weekend

[rapid hedge]

thanks a bunch 🙏

OOOOH you know what

bet this is because the audio I'm trying is playing for less than 2 secs

(= the impulse duration)

update: bet the loud BUMP is because we never check if source_transform == listener_transform

there's a null vec in there

hmm ffmpeg tells me that caw.ogg is 2160 ms long

but it crashes the same way

BUT

selfless_courage.ogg does not crash

setting the impulse duration to 1 sec makes it take longer to crash

but also introduces weird audio glitches between recordings

[slate scarab]

hm, i don't think there should be any correlation between the IR and the actual length of the sample inputs (the only thing that shorter samples will do is present more zeroes basically)

[rapid hedge]

setting the time between playbacks a little bit higher also makes it more stable

[rapid hedge]

weird

[slate scarab]

but idk i guess we'll have to do some spelunking to figure it out

[rapid hedge]

it could be correlated insofar as removing a source could be a crash

[slate scarab]

ya i guess there's a lot going on there

In other words, there could maybe be a resource that's a shared pointer produced by the source's simulation outputs that gets dropped when the source is removed or something?

I guess that shouldn't happen if they're shared pointers, but... idk. Might be worth checking what removing a source actually does.

[rapid hedge]

Setting the sample player to loop makes it take longer to crash, but also spawns many sources

setting it to loop and spawning exactly as many sources as the maximum (8 by default, -1 for the source at the listener = 7) makes it not crash, but also not play audio

[slate scarab]

Making it loop spawns a bunch of sources?

[rapid hedge]

the test I wrote plays an effect every 200 ms

[slate scarab]

i see

[rapid hedge]

I added some docs to my bug repro code

That can give you an idea of what I investigated

[rapid hedge]

Update: yep, that works much better! Thanks!

@slate scarab huh, turns out there is no direct audio in general at all when the audio source is straight forward

Is binaural stuff defined like that?

[slate scarab]

no, there should just be a little bit of filtering going on (to simulate how a front-facing sound strikes your face and torso)

[rapid hedge]

There's a spatial blend param that I have at 1.0, but fiddling it down doesn't change anything about the sound being inaudible

[slate scarab]

maybe that's why it has sounded (subjectively, to me) really overly muffled for directly forwards sounds

[rapid hedge]

Oh for fucks sake

it's another magical property on the effect params that the simulation output does not set

and the default is 0

whatever 0 is

but setting the directivity to 1.0 fixes it

Oh I know

I bet that's how far around the object the sound goes

and since I spawn the listener and the source both with a default rotation...

the source points directly away from the listener

and if there's no reflection, the sound will just directly shoot away

cool beanz

[slate scarab]

so 0 would be extremely directed sound?

[rapid hedge]

That's my hypothesis

AHA

Found the culprit

this is an audionimbus issue!

See that Some(params.directivity)?

When that is manually overwritten to None it becomes omnidirectional

So Some(1.0) == None AFAICT

because...

...of this line when it gets translated back to the FFI format

let's investigate the C++ source to see if my ideas are right

[slate scarab]

ah, so it's been on the whole time? I do wonder what a directivity of 0 means

[rapid hedge]

because if they are, that would mean that our occlusion was potentially also borked

[slate scarab]

my intuition is that it wouldn't interact with the directivity too much, but

[rapid hedge]

the occlusion is also set wrongly, BUT the value ends up being the same as if no value is set

so our occlusion is correct lol

[slate scarab]

oh i see

[rapid hedge]

I think Valve was not sure either how their information flows

Because the Unity and Unreal plugins end up overwriting everything they can away from any defaults, just to be sure

well, let's do that too

See, audionimbus tries to be nice by going "Oh, you have a valid number here, let's enable the flag for actually reading that number"

but Unity Steam Audio goes "Please manually enable any flags you care about since the numbers might be meaningless, thanks"

[past sandal]

Good to hear 🙂

I think this directivity is the resulting gain from a directivity calculation btw, which takes in a dipoleWeight and dipolePower

Even if you're not using FMOD it's probably worth it to install it together with the Steam Audio plugin, so you can play around with those parameters and see how it will affect the sound (it has a visualizer)

[rapid hedge]

yep I added that now

my issue is now the listener's rotation seems to be ignored

the source's rotation is correct

OOOOh it needs to be in local space

[rapid hedge]

Alright, fixed the directivity stuff

not the crash yet

but the directivity is much better now on main if you want to try 🙂

Wonder if it still feels muffled from the front now @slate scarab.

[rapid hedge]

All processors run exclusively in Last, right?

hmm these systems run in PostUpdate but have no way of ordering against them

And some of our stuff also runs in PostUpdate. Wonder if that could lead to a race condition?

Can't really move it to Last either since some seedling stuff is not fully ordered, like this

But that's probably not the cause of the segfault. I tried setting all our systems in Update and I still get it.