bevy_seedling

[ionic sedge]

Like reverb.

[short fossil]

To elaborate; since I never heard of an aux send, I was thinking of it like a package I send away, meaning that the original place now has no signal anymore

So like, I want to send it to nodes A B and C, and I want each node to get a different volume?

Did I get that right?

[ionic sedge]

Well it can only have one (additional) routing destination, so it's like if you wanted sampler 1, 2, and 3 to go to node A, where each one can have a different volume.

(In this scenario, each sampler is chained into a SendNode whose destination is A.)

[short fossil]

Is that the same as chaining each sampler to a volume node followed by the bus?

[ionic sedge]

Or rather, additionally connecting each sampler to a VolumeNode, and then that VolumeNode to A?

[short fossil]

Yeah exactly

[ionic sedge]

this would be so much easier to communicate with a node graph UI 😅

yeah that's pretty much what it does

[lapis stone]

okay, one last q on this sorry 😅 im running into one last issue where i need to implement RealtimeClone for something to be a node parameter, but that would be false given id need to clone the whole buffer of stuff in the fftconvolver 🤔 but maybe i have my wrappers in the wrong spots or something

[static quest]

Oh, that's right, we can't exactly use the automatic parameter patching system for an OwnedGc. It has to be a NodeEventType::Custom.

[ionic sedge]

which imo is okay -- if you need this functionality, you ought to just send that message manually i think

(Or at least, it would be fine with bevy_seedling, but of course that's not the only scenario.)

[lapis stone]

is there an example of how to use this somewhere in another node? im not really sure what this entails to be honest

[static quest]

There isn't an example, but in the custom Diff implementation you would use EventQueue::push instead of EventQueue::push_param.

And in the node processor, you would use ProcEvents::drain instead of ProcEvents::drain_patches.

But yeah, it probably would be helpful to make an example.

[slim pulsar]

what a fucking amazing reference project u is, wow

very good night time reading

zzz

[lapis stone]

oh okay i didnt realize this just meant writing a custom diff impl, that sounds straightforward

i think i confused something in the macro with Diff also requiring RealtimeClone but i see that is not the case

[lapis stone]

cool managed to get that working after some trial and error just need to cleanly fade/declick between both IRs now

i also have the delay/echo more or less done besides preserving pitch when changing time (which is actually probably gonna be the most time consuming bit...)

[static quest]

Oh yeah, smoothly changing time is a notoriously hard problem. (Though I see someone linked you a paper on the Rust Audio discord server.)

[obsidian tusk]

I don’t know if you saw but StaffPad open-sourced their time-stretching algorithm so it could be integrated into Audacity https://github.com/audacity/audacity/tree/master/au3/libraries/lib-time-and-pitch/StaffPad

It’s the only standalone open-source time stretching algorithm I’m aware of, the only other one I know is in Tracktion Engine and it’s super reliant on JUCE

Like, as far as production-grade algorithms go, obviously there’s a bunch of simpler ones that are public domain. I was considering doing a Rust rewrite because it feels like it’d be a great contribution to the Rust audio space but I’ve got a lot of stuff to do at work and on my main personal programming project and I’m trying not to let myself get too sidetracked 😅

I haven’t dug into it too much though so I can’t vouch for it yet

[static quest]

Yo, sick!!! I'm definitely using this in my DAW project.

[static quest]

There's Rubberband and SignalSmith stretch.

[obsidian tusk]

Ooh just had a look and rubberband's been updated since I last saw it a couple of years ago, it was pretty gnarly when I saw it before. SignalSmith I don’t know, I’ll take a look

Hm took another look and rubberband doesn’t seem as bad as I remember, I don’t know why I dismissed it before. Still pretty limited (on purpose, it looks like) but seems like it does the job

[heady robin]

@ionic sedge is there a bevy 0.17 branch for seedling? Can't find one

[ionic sedge]

https://github.com/CorvusPrudens/bevy_seedling/tree/v0.6.0 represents the latest work, although it's very much in flux. 0.6.0-rc.1 is available on crates.io at the moment if you want something more stable (and semver compatible with 0.17).

Although I'm not sure the RC version includes all the fixes from 0.5 either way, I'm really hoping to get everything organized this weekend for a proper 0.17 release.

[heady robin]

Alright, I'll move to 0.16.0-rc.1 and hope not to get the index out of bounds thing

Thanks!

[ionic sedge]

If you do run into errors, feel free to try out the branch I linked. While it's not necessarily "done," it does include those fixes and shouldn't have any further major breaking changes.

[heady robin]

I can't find an rc.1 branch

[ionic sedge]

the rc is on crates io

[heady robin]

ohhh I see

[ionic sedge]

I didn't tag it for some reason.

I could probably find it retroactively.

[heady robin]

But like, I should be able to get to it directly as a crate version instead of routing to a branch

[ionic sedge]

Yes, like version = "0.6.0-rc.1"

[heady robin]

In case other people would read and want to try out-
You also need to set firewheel-core = "0.8.0-rc.1"

[slim pulsar]

hey, ik this isn't seedling related necessarily, but i am surprised that firewheel has a full event queue on startup

I'm not exactly sure why this is. who be sending firewheel stuff

but no audio is playing anyway.

[ionic sedge]

Is this a vanilla setup, or have you added nodes of your own or anything like that?

imo that shouldn't be an error, actually -- it's really a warning

But we could probably stand to increase the base capacity again.

[slim pulsar]

kinda need a sanity check on this https://github.com/CorvusPrudens/bevy_seedling/issues/67
tl;dr: sampler is despawned when setting playback to the first frame after playback has started. looking at the code, I'm guessing the logic is

Playhead set to 0, paused -> SamplerNode despawns? -> SamplerOf despawns, despawning Sampler

well, that can't be right. the remove hook doesn't do that

[slim pulsar]

yeah, can't find any code that calls .remove::<SamplerOf>, not sure what to do

[ionic sedge]

Hm, I’ll double check if this occurs with the latest branch (oh I see you patched it).

[heady robin]

nooo

[ionic sedge]

I addressed this on the issue. The latest commit should resolve the underlying issue here.

[ionic sedge]

Yeah, that fix wasn't forward-ported. I'm working on the new release now which should be out in a day or two.

[slim pulsar]

oh wow, thank you!

I really was lookin for the sucker for a solid half hour haha, i did give it some time!

[ionic sedge]

i should probably add some testing for this behavior it has changed a couple times so far

[slim pulsar]

oh crap

i feel like "stop" whatever it is

is pause and Frames(0). I cannot find it anywhere

but i swear i saw it somewhere

[ionic sedge]

it gone

[slim pulsar]

o ok

[ionic sedge]

stop is now, well, despawn

[slim pulsar]

yay1!

y'all are awesome i like that a lot

[ionic sedge]

(it would trigger the PlaybackCompletionEvent previously)

but we could add it back with the semantics you're expecting if you think that's useful

[slim pulsar]

oh no im good

you're awesome, thanks

if you're ever in atlanta, i owe you a beer

[ionic sedge]

oki if i visit i'll take you up on it :)

[short fossil]

Alright, I'm now at the step where I need to process the audio sources. My plan is

• Do all processing in the ECS
• Only process sources that can influence the NPC (do a simple distance falloff check)
• To process a source, I need some information from the audio thread:
  • The last N samples of audio (resampled)
  • The loudness

I thought of writing a little SummaryNode or something like that that calculates that information in the audio thread so that the ECS can just read it

And I'm looking at the LoudnessNode for inspiration

Some questions I need help with:

• Some NPCs need to hear every 200 ms, some every 500 ms (depending on player distance). Would something "bad" happen if I set my frame size to capture the last 500 ms but then process it every 200 ms?
• Should I just use the LoudnessNode for the loudness part, or should I recreate just the part I need?
• Does it need to know about the fact that the NPCs get all audio downsampled?
• What's a LUF?
• Why does the EbuR128 specifically use 400 ms as momentary? Is that a standard?
• Is EbuR128 public anywhere?
• Does it matter if I read the momentary 400 ms loudness every 200 ms / every 500 ms?

@ionic sedge in case you happen to have time 🙂

Oh another one:

• For the information about what the last N samples were, is it fine if I just take mono input in my node? I think that doesn't divide by the number of channels, right? Should I? We do divide that in the Steam Audio effects that require mono (following Unity and Unreal's example), so I feel like this node here should too?

[ionic sedge]

Some NPCs need to hear every 200 ms, some every 500 ms
Hm, I wonder if you could create two "pools" here

Should I just use the LoudnessNode
Definitely overkill -- you'll probably want the RMS (root-mean squared) amplitude. Just take each sample, square it, take the average of the whole set, and then get the square root. If you want, you can then convert this to decibels (which we do have methods for in Firewheel) or you can just operate on the value itself.

Does it need to know about the fact that the NPCs get all audio downsampled?
nah, you lose some "resolution" when downsampling, but the amplitude is still the amplitude

I wouldn't worry about the LUFs given the above.

The various times given by the EBUR stuff are just heuristics for evaluating loudness. You don't need to worry about this here because you want to know the average loudness of a block of samples, not any particular time really.

[short fossil]

ah cool!

[ionic sedge]

For the information about what the last N samples were, is it fine if I just take mono input in my node? I think that doesn't divide by the number of channels, right?
Yeah it just sums it, so it does mess with the loudness a bit.

[short fossil]

Alright, then I'll downmix

[short fossil]

Hmm I could let the summary node store both the 200 ms and the 500 ms windows

ah heck

the simulator itself also wants the frame size

so two simulators

that's probably fine though

The expensive part is probably in the scene, which stores the BVH

and I can share that

[ionic sedge]

or maybe just check the last 200ms for the further away NPCs maybe? like skip every other period?

[short fossil]

hmm yeah that could work

I'm just wondering about the implications

[ionic sedge]

maybe that would feel random though for players

[short fossil]

Since it seems to me like it's not significantly more computation / memory to just store the 500 ms data

just a bit annoying really

[ionic sedge]

If you're transferring samples from the audio graph to the ECS though, you could always just store the last 200ms as well.

[short fossil]

what do you mean?

[ionic sedge]

i guess you still run into issues if you want to simulate over two blocks instead of one

[short fossil]

(there are only max about 10 ai-audible sources at a time)

[short fossil]

hmm fair

[short fossil]

(usually more like 2)

Also, my SO had neat idea for eliminating one source: treat the left and right footstep sounds as a single audio

from seedling's perspective, it's two audio players

but the NPC doesn't need to respect that

[short fossil]

@ionic sedge do you think FixedProcessBlock will work with an output size of 0?

given that I never want to actually write anything from the summary node

[ionic sedge]

as in no output channels

[short fossil]

yep

or well

technically 0 output_channels and 0 max_output_size

Or I wonder if I could skip on the fixed process block if I just use a sliding window

[ionic sedge]

It's supposed to work if there's either no inputs or no outputs, at least.

[short fossil]

What does the fixed block do exactly?

• wait until we have a full block
• use that block
• then what? is the block yeeted or is it slid back?

[ionic sedge]

Any time the inputs reach capacity, the block is processed and pushed to the output. It drains the outputs if they're larger than the (outer) output buffer size. If they're empty, then it'll just never run that code.

[short fossil]

Ah gotcha

[ionic sedge]

The inputs will "reach capacity" whether there are input channels or not -- it tracks that as a length parameter.

[short fossil]

So if I need inputs for the last 200 ms and for the last 500 ms, should I use two fixed blocks?

[ionic sedge]

ya you could do that

although we could really use a "reborrow" method on the proc buffers 😅

[short fossil]

was just thinking the same haha

beginner question

downmixing is just
downmixing[sample] = (input[0][sample] + input[1][sample]) / 2,
right?

[ionic sedge]

ya

[short fossil]

thx

I was wondering because the C++ code looks more intimidating than that

void AudioBuffer::downmix(const AudioBuffer& in,
                          AudioBuffer& out)
{
    assert(in.numSamples() == out.numSamples());
    assert(out.numChannels() == 1);

    memcpy(out[0], in[0], in.numSamples() * sizeof(float));
    for (auto i = 1; i < in.numChannels(); ++i)
    {
        ArrayMath::add(in.numSamples(), in[i], out[0], out[0]);
    }

    ArrayMath::scale(in.numSamples(), out[0], 1.0f / in.numChannels(), out[0]);
}

And I suspected that was just because it has a lot of boilerplate haha

[ionic sedge]

ya cause of the variable number of channels

[short fossil]

Does this seem reasonable?

impl AudioNodeProcessor for InputBufferProcessor {
    fn process(
        &mut self,
        proc_info: &ProcInfo,
        proc_buffers: ProcBuffers,
        events: &mut ProcEvents,
        extra: &mut ProcExtra,
    ) -> firewheel::node::ProcessStatus {
        if proc_info.in_silence_mask.all_channels_silent(2) {
            return ProcessStatus::Bypass;
        }
        let temp_proc = ProcBuffers {
            inputs: proc_buffers.inputs,
            outputs: proc_buffers.outputs,
        };
        let scratch = extra.scratch_buffers.first_mut();
        self.near_block
            .process(temp_proc, proc_info, |inputs, _outputs| {
                let scratch = &mut scratch[..FRAME_SIZE_NEAR as usize];
                for (i, sample) in scratch.iter_mut().enumerate() {
                    *sample = (inputs[0][i] + inputs[1][i]) / 2.0;
                }
                for (src, dst) in scratch.iter().zip(self.state.input_near.iter()) {
                    dst.store(*src, Ordering::Relaxed);
                }
                self.state
                    .loudness_near
                    .store(rms(&scratch), Ordering::Relaxed);
            });
        let temp_proc = ProcBuffers {
            inputs: proc_buffers.inputs,
            outputs: proc_buffers.outputs,
        };
        self.far_block
            .process(temp_proc, proc_info, |inputs, _outputs| {
                let scratch = &mut scratch[..FRAME_SIZE_FAR as usize];
                for (i, sample) in scratch.iter_mut().enumerate() {
                    *sample = (inputs[0][i] + inputs[1][i]) / 2.0;
                }
                for (src, dst) in scratch.iter().zip(self.state.input_far.iter()) {
                    dst.store(*src, Ordering::Relaxed);
                }
                self.state
                    .loudness_far
                    .store(rms(&scratch), Ordering::Relaxed);
            });

        ProcessStatus::Bypass
    }
}

[ionic sedge]

Oh you have a buffer of atomics? That might be slow

You might take a look at Firewheel's stream reader node https://docs.rs/firewheel-nodes/0.8.0-rc.1/src/firewheel_nodes/stream/reader.rs.html -- it uses basically a channel for reading back from the graph.

[short fossil]

Oh I just blindly copied that from the loudness lol

I don't really need it

[ionic sedge]

well I mean it's an audio buffer, right?

[short fossil]

yep

I mean I don't need it to be all atomic floats necessarily

[ionic sedge]

ya it might be expensive to write atomically to hundreds of samples
but i mean im sure it'll work

[short fossil]

yeah that seems better

thx!

[short fossil]

How come the loudness node does it?

Oh wait that is an array of channels

not of samples

aaaah

[ionic sedge]

ya

[short fossil]

okay, makes sense

[short fossil]

is there anything I need to take into account for the channel of choice?

Or can I use trusty old crossbeam_channel?

[ionic sedge]

well if you want to be tidy you'd need to make sure it won't make syscalls (under normal conditions)

[short fossil]

Hmm I see

[ionic sedge]

which is part of why billydm made that resampling channel

[short fossil]

I saw Billy's resampling_channel, yeah

I'm just a bit confused how to hold that one

But should I be using that one?

[ionic sedge]

if you can get a handle on it it would be a solid way to go

[short fossil]

I'm mostly feeling overwhelmed by

• ResamplingChannelConfig and
• the fact that I need to recreate it when the sampling rate changes

[ionic sedge]

Ah yeah I see, there's a lot of cruft you don't need. Internally it uses a ring buffer from ringbuf.

That's probably a better choice over a channel assuming you can read it frequently enough from the ECS (which should be a good assumption, since you'd only read every 200-500 milliseconds).

The ringbuf ring buffer is single-producer, single-consumer, which is great for this use case.

If you give the ring buffer twice the capacity that you want to transfer, then you should have no problem keeping up. So that would be 400ms and 1s of downsampled audio respectively.

Although if you're constantly sipping samples each frame from the ECS, then you can get away with a smaller buffer.

This approach is a touch more annoying, but it's fast and guarantees no locks or syscalls.

Like a channel, it has a producer and consumer, so that aspect should be familiar at least.

[short fossil]

Well if I have enough NPCs that number goes up to reading every frame

It's just every individual NPC that takes 200 ms, but I'll stagger them

[short fossil]

neat! Let me try

[ionic sedge]

To be clear, you can skip the fixed processor block with this approach. Just write directly to the ring buffer.

[short fossil]

oh I didn't realize 👀

also, how would you give the consumer to the ECS?

Store it in the InnerState?

[ionic sedge]

create the channel post-insertion and send the producer to the node as an event

[short fossil]

Ah so like for the reader node

[short fossil]

So e.g. in an On<Add> observer for the node, right?

[ionic sedge]

Actually I think a system would be better -- you could add it to the Last schedule after SeedlingSystems::Acquire.

Well, unless that node isn't a part of the pools. Then it doesn't really matter.

[ionic sedge]

But to clarify on this point -- you can easily tell if a node is "real" by using a system:

fn establish_channel(
    nodes: Query<&mut AudioEvents, (Added<FirewheelNode>, With<NpcReaderNode>)>,
) {
    // ...
}

[short fossil]

thanks! I was about to ask haha

[ionic sedge]

and inserting it where i mentioned

Any node with FirewheelNode is necessarily real and in the graph.

[short fossil]

I would have added those sources to the regular Steam Audio Pool and an AiSimulationPool

[ionic sedge]

I guess Changed is better actually, since you might need to recreate the channel if the node is recreated.

[short fossil]

And I suppose I can store the consumer (behind a Mutex) on a component on the same entity, right?

[ionic sedge]

oh does it need to be in a mutex?

[short fossil]

Yeah the consumer is not thread safe

The reader node also stores a mutex

but in the ActiveState

[ionic sedge]

i.e. it's Send, !Sync

ya that's fine

no need for the Arc in this case since it can just live on the entity

[short fossil]

If I store it on the same entity, how do I get it when iterating over the "fake" nodes? Idk the correct terminology here

Is that Follows?

[ionic sedge]

Do you need to iterate over those? I don't think they matter.

[short fossil]

Well in the end I iterate over NPCs

And they need to iterate over all nodes that are "relevant"

for that I was planning on looking at their distance to the NPC and their volume

to get their distance, I can just iterate over the SamplePlayers that are tagged with AiAudible, easy, those have a Transform

But then I also need the relevant input buffer nodes

[ionic sedge]

And how do you plan to hook those buffers up?

That's the key I suppose.

[short fossil]

excellent question

brb

[ionic sedge]

i'd really like relationship-aware query observers

fn add_buffer(
    data: On<DataAdded<
        Entity, 
        (With<SamplerNode>, With<Ancestor<SteamAudioPool>>),
    >>,
    mut commands: Commands
) {
    commands.entity(*data).chain_node(NpcListenerNode::default());
}

imagine

anyway I'd probably basically do exactly this ^ but in a system

or you could do it in an observer for every SamplerNode I guess, it's not like they're created very often

[short fossil]

yeah that's what I thought too

[ionic sedge]

Doing it this way, you'd probably want to establish some link between the listener and the actual sampler node. You could make it a child of the sampler node I suppose. Then, to find it, you'd traverse the Sampler relationship on the SamplePlayer, then look through the sampler's children for your listener.

[short fossil]

Hmmmmmm

So I already have this though mut nodes: Query<(Entity, &mut AudioEvents), (Changed<FirewheelNode>, With<InputBufferNode>)>,

And that's fine

But I set the InputBufferNode up as children of samplers

SamplerNode

and those SamplerNodes are... children? Of the pool

[ionic sedge]

ya

[short fossil]

Is the pool also split into a fake and real?

[ionic sedge]

Damn, actually they're linked by a private relationship.

[short fossil]

hecc

[ionic sedge]

No, there's no need for a template SamplerNode in bevy_seedling.

[short fossil]

NGL sometimes I feel like this: https://www.youtube.com/watch?v=EvKTOHVGNbg

haha

[ionic sedge]

I suppose they could simply be children of the pool.

Or at the very least I could make the relationship public. I think I was (legitimately) worried about overloading of children.

It matters for the simplicity of the pool code (which is already a mess) that each related entity is actually a sampler.

Either way, I think you should be able to do this.

The question is which way is better.

[short fossil]

Relationship between what and what?

You mean the Sampler?

[ionic sedge]

okay now you sound like king of the hill guy

hwat

[short fossil]

hwat

[ionic sedge]

Each SamplerNode and the pool they belong to.

[short fossil]

Do I look like I know hwat a pool sampler ringbuf state event node is? I just want a buffer of my god-dang input

[ionic sedge]

This is the private relationship that links the sampler nodes to their pool.

[short fossil]

that would indeed be nice to have

but also

[ionic sedge]

I was worried about semantic overlap with the Sampler relationship.

in people's minds

[short fossil]

Can't I just do the good old

#[derive(PoolLabel, Reflect, PartialEq, Eq, Debug, Hash, Clone)]
#[reflect(Component)]
struct AiPool;

fn init_pool(mut commands: Commands) {
    commands.spawn((
        Name::new("AI Sampler Pool"),
        SamplerPool(AiPool),
        sample_effects![InputBufferNode],
    ));
}

?

[ionic sedge]

sure if you want to play samples twice, which is fine

[short fossil]

oh that will play them twice?

[ionic sedge]

no, but how is that getting to the player's ears?

[short fossil]

can a sample not belong to two pools at the same time?

[ionic sedge]

no

[short fossil]

oh

Well, that changes things

Okay hear me out

[ionic sedge]

I think the lifecycle management would get a bit confusing.

[short fossil]

what about

[ionic sedge]

lay it on me

[short fossil]

#[derive(PoolLabel, Reflect, PartialEq, Eq, Debug, Hash, Clone)]
#[reflect(Component)]
struct AiPool;

fn init_pool(mut commands: Commands) {
    commands.spawn((
        Name::new("Music audio sampler pool"),
        SamplerPool(AiPool),
        sample_effects![
            InputBufferNode, // Passes the input directly to the output
            SteamAudioNode
        ],
    ));
}

And I make sure that my ai-audible samples are always in this pool

[ionic sedge]

ya you could do that

the other system would still work as-is (that establishes the producer and consumer)

[short fossil]

Then when the input buffer is added on an entity that also has EffectOf, I create the ringbuf, send the producer to the buffer node, and spawn the component that contains the consumer

Does that make sense?

[ionic sedge]

no the ringbuffer should only be created for the entity with FirewheelNode, which is mutually exclusive with EffectOf

Another way to think of it is that you only need to create the ringbuf pair once.

You don't need to do it every time you spawn a SamplePlayer.

That's what this would do for you mut nodes: Query<(Entity, &mut AudioEvents), (Changed<FirewheelNode>, With<InputBufferNode>)>

[short fossil]

I don't? I thought that every sample player would want to buffer their inputs

so that the AI can read the current "output" of any given sample player

Oh off-topic, I don't think we need the channels setup anymore in bevy_steam_audio:

pub(crate) fn setup_nodes(mut commands: Commands, quality: Res<SteamAudioQuality>) {
    commands.spawn((
        SamplerPool(SteamAudioPool),
        VolumeNodeConfig {
            channels: NonZeroChannelCount::new(quality.num_channels()).unwrap(),
        },
        sample_effects![SteamAudioNode::default()],
    ));
}

Since SteamAudioNode takes and outputs stereo

[ionic sedge]

Yes for this node it's not necessary

[ionic sedge]

A SamplePlayer merely asks for a slot in the pool. The actual nodes are always there, either active or idle. So to find the consumer, you'd just jump through a couple relationships from the SamplePlayer entity. You don't need to create a new pair every time a SamplePlayer acquires a slot.

[short fossil]

I see

[ionic sedge]

That's where the Followers come in as you noted eariler.

[short fossil]

But if the buffer exists only once, isn't it overwritten all the time by whatever sample player happens to get the slot this time?

[ionic sedge]

you can think of it like (pseudocode)

(
    SamplePlayer,
    sample_effects![(
        // template node
        InputBufferNode, 
        followers![(
            // real node
            InputBufferNode,
            Mutex<Consumer>,
        )],
    )],
)

[ionic sedge]

I would frame it as "the ringbuffer may contain data from previous samples played over the last 200 or 500ms period"

Indeed, if you want to avoid that, you could clear it every time a new SamplePlayer is assigned.

[short fossil]

that is helpful, thanks!

[short fossil]

I'm just thinking that if I want to filter sample players data before processing by "is this sample player at this location playing loud enough for this NPC at this other location to hear it" it would be bad if that data contained stuff from sample players at other locations 👀

Or am I fundamentally misunderstanding something?

[ionic sedge]

No I think you're correct. It would be easier to manage if you create a new pair for every player. It's a simple process.

However, this has reframed my thinking a bit.

[short fossil]

oh, how so? 🙂

[ionic sedge]

Each SamplePlayer that you want NPCs to listen to should have a VecDeque of the last 500ms of (downsampled) audio. When you spawn these SamplePlayers, they should establish a ringbuf pair, and consume the samples from the ringbuf every frame.

Any listening NPCs can then grab either the last 200ms or 500ms of audio. The sample may have been played only 100ms ago, but that's fine. Any samples before 100ms would just be considered silence.

[short fossil]

Smart 👀

The VecDeque is a sliding window, but no NPC will ever notice that, as they are activated in a delayed way

#[derive(PoolLabel, Reflect, PartialEq, Eq, Debug, Hash, Clone)]
#[reflect(Component)]
struct AiPool;

fn init_pool(mut commands: Commands) {
    commands.spawn((
        Name::new("Music audio sampler pool"),
        SamplerPool(AiPool),
        sample_effects![InputBufferNode, SteamAudioNode::default(),],
    ));
}

fn establish_channel(
    mut nodes: Query<(Entity, &mut AudioEvents), (Changed<InputBufferNode>, With<EffectOf>)>,
    mut commands: Commands,
) {
    for (entity, mut events) in nodes.iter_mut() {
        let (prod_near, cons_near) = HeapRb::new(FRAME_SIZE_NEAR as usize).split();
        let (prod_far, cons_far) = HeapRb::new(FRAME_SIZE_FAR as usize).split();
        let event = InputBufferInitEvent {
            near: prod_near,
            far: prod_far,
        };
        events.push(NodeEventType::custom(event));
        commands.entity(entity).insert(InputBuffer {
            near: [0.0; FRAME_SIZE_NEAR as usize],
            far: [0.0; FRAME_SIZE_FAR as usize],
            inner: Mutex::new(InputBufferConsumers {
                near: cons_near,
                far: cons_far,
            }),
        });
    }
}
struct InputBufferInitEvent {
    near: <HeapRb<f32> as Split>::Prod,
    far: <HeapRb<f32> as Split>::Prod,
}

#[derive(Component)]
pub(crate) struct InputBuffer {
    // the `near` and `far` buffers are updated once per thread
    near: [f32; FRAME_SIZE_NEAR as usize],
    far: [f32; FRAME_SIZE_FAR as usize],
    inner: Mutex<InputBufferConsumers>,
}

struct InputBufferConsumers {
    near: <HeapRb<f32> as Split>::Cons,
    far: <HeapRb<f32> as Split>::Cons,
}

I have this here right now

but the near and fars should be VecDeques

and we only need one buffer as you say

[ionic sedge]

no you should still use ringbuf to communicate between the ECS and audio graph -- but each frame, you drain all the samples from the ringbuf into the VecDeque (and pop off any extra if it's over 500ms)

oh, well yeah you mean for InputBuffer

[short fossil]

yeah I mean I only need one ringbuf, not two

Got it:

#[derive(PoolLabel, Reflect, PartialEq, Eq, Debug, Hash, Clone)]
#[reflect(Component)]
struct AiPool;

fn init_pool(mut commands: Commands) {
    commands.spawn((
        Name::new("Music audio sampler pool"),
        SamplerPool(AiPool),
        sample_effects![InputBufferNode, SteamAudioNode::default(),],
    ));
}

fn establish_channel(
    mut nodes: Query<(Entity, &mut AudioEvents), (Changed<InputBufferNode>, With<EffectOf>)>,
    mut commands: Commands,
) {
    for (entity, mut events) in nodes.iter_mut() {
        let (prod, cons) = HeapRb::new(FRAME_SIZE_FAR as usize).split();
        let event = InputBufferInitEvent(prod);
        events.push(NodeEventType::custom(event));
        commands.entity(entity).insert(InputBuffer {
            buff: VecDeque::with_capacity(FRAME_SIZE_FAR as usize),
            cons: Mutex::new(cons),
        });
    }
}
struct InputBufferInitEvent(<HeapRb<f32> as Split>::Prod);

#[derive(Component)]
pub(crate) struct InputBuffer {
    buff: VecDeque<f32>,
    cons: Mutex<<HeapRb<f32> as Split>::Cons>,
}

[ionic sedge]

ya

[short fossil]

does this seem sensible?

yay!

From bevy_seedling's perspective, what's the right schedule for these?

Namely for

• establish_channel and
• update_buffer (which updates the deque by draining the consumer)

I assume Last, and then relative to some system set that seedling exposes?

[ionic sedge]

Hm, I think what I'd do is adjust establish_channel slightly. It would trigger for the pool's template too, which is slightly awkward. It might cause a problem?

But I'd run it for every new SamplePlayer that has a related InputBufferNode. This system should run after the pool normalizes the effects in SeedlingSystems::Pool. So I'd place it in the SeedlingSystems::Queue set.

[short fossil]

oh right, good catch

I could simply check if the parent contains SamplerPool<AiPool> and if so skip it

[ionic sedge]

or conversely check if it has AiPool and do it

Or if you want it to work in any pool, you could also check for PoolLabelContainer, which is a type-erased container placed on the SamplePlayers. nvm that's also on the pool

[short fossil]

Any specific place I should put the update_buffer system?

I don't think it matters much

since the AI is not run after Last anyways

So I guess PreUpdate is good

[ionic sedge]

ya doesn’t really matter

[short fossil]

Alright, another little issue (if you have time)

no pressure if not haha

you've been extremely helpful already!

I'm trying to add the resampler on top, https://docs.rs/rubato/latest/rubato/struct.FastFixedOut.html

and since that bad boy allocates on new, I want to create it as a field of the processor

But since we no longer use the fixed block abstraction, I don't know what to use for the chunk_size 👀

Do I use max_block_frames and then just only read the first n samples when resampling?

[ionic sedge]

hmmmmm

well you could do a fixed processor i suppose

if it makes the resampling easier

[short fossil]

ngl this description is scary

[short fossil]

do you happen to know a place in seedling or firewheel where I could yoink some boilerplate for that?

[ionic sedge]

you don’t gotta deal with allat i think

or at least basically everything after the first five steps doesn’t matter

since you’re just writing into the ring buffer

you don’t need to process partial if the size is always exactly what you need

like with the fixed processor

but no I don’t think there’s any clear example of this anywhere

[short fossil]

wheew, good

[short fossil]

Alright, got something

Mind giving it a look when you happen to have time? https://github.com/janhohenheim/thief_sense_demo/blob/b7c1424f9c48b8d0af9fdea1e5c6eafa8376205c/src/demo/ai/hearing/node.rs

The mono input is now buffered like this:

for (i, sample) in self.resample_in[0].iter_mut().enumerate() {
    *sample = (inputs[0][i] + inputs[1][i]) / 2.0;
}
let delay = self.resampler.output_delay();
let new_length =
    (FRAME_SIZE_FAR * SAMPLING_RATE / proc_info.sample_rate.get()) as usize;
self.resampler
    .process_into_buffer(&mut self.resample_in, &mut self.resample_out, None)
    .unwrap();
prod.push_slice(&self.resample_out[0][delay..(delay + new_length)]);

I hope that's right

Because I'm really not sure how I would verify it

[ionic sedge]

ya that looks right

[short fossil]

heck yeah

[ionic sedge]

you could verify it by listening to the downsampled audio haha

[short fossil]

heh, fair enough

[ionic sedge]

write them bytes into a wav file or something and see how it sounds
it'll be super obvious if there are seams or any artifacts like that

[short fossil]

what is a seam in audio?

[ionic sedge]

you know like a pop or something

[short fossil]

ah gotcha

[short fossil]

I should be able to just populate the output buffer in the fixed block with the downsampled data, right?

[ionic sedge]

actually i guess it's a little weird since you want to pass the normal audio through

You'd want to strip out the actual output buffers from the ProcBuffers that you give to the fixed block.

oh you set it to bypass

Anyway, there's no reason really to write to the fixed block outputs. Allocating output buffers there seems wasteful for no benefit.

[ionic sedge]

oh you mean if you want to see how it sounds

no it wouldn't work right because there would be way fewer samples produced, so it would constantly run out

[short fossil]

Oh ok

Then write them directly into a file?

[ionic sedge]

ya, conveniently you can do that from the ECS once it's all set up

[short fossil]

oh right!!

[short fossil]

this is good right

[short fossil]

okay fixed that

thread 'cpal_alsa_out' (256192) panicked at src/demo/ai/hearing/node.rs:215:18:
called `Result::unwrap()` on an `Err` value: Insufficient buffer size 4000 for input channel 0, expected 44104

so I suppose rubato doesn't want the input and output buffers to have the same len?

It's happening in

self.resampler
    .process_into_buffer(&mut self.resample_in, &mut self.resample_out, None)
    .unwrap();

[ionic sedge]

correct, because you're resampling

[short fossil]

oooooh OFC

So I need the input to be massively bigger

got it

[ionic sedge]

although resampling a whole second worth of audio all at once might be a lot you could probably make the fixed block a lot smaller and only write small bits at a time to the ring buffer

the ECS-side VecDeque means you'd don't need any kind of tight synchronization or anything

you can just write (and read) the bytes as they come

[short fossil]

well here's my beautiful 68 byte wav

alright, enough for today

thanks for all the help again

You're carrying this NPC hearing thing haha

[stoic igloo]

LUFS is a new audio standard.
Before, it would catch the highest ocean wave (peak) and using beach as an example. Now, LUFS measures how high the tide is to decide to lower the volume to maintain some Audio Act to quiet noise pollution (Loudness War) It is a thing still, just less now. It is an average. (tide)

[obsidian tusk]

I had a look and the earliest definition of LUFS in the modern understanding is only from 2007, pretty cool. I figured it would’ve been older

[static quest]

What are you trying to do exactly?

If all you want to do is measure the loudness of a signal, the easiest way is to measure it in the nodes's processing method, and then store the result in an AtomicFloat if the new value is greater than the one stored in the atomic. Then in the game loop, you just poll the AtomicFloat for the latest value and swap it with zero. (This is how the peak meter node does it.)

It would probably also make sense to have an RMS/LUFS node as a factory node.

Though in the context of games, LUFS is probably overkill. RMS is good enough if all you want to do is have your game react to the player's voice and whatnot.

LUFS is more for measuring the perceived volume of a signal over a broad period of time, where as RMS is better for low latency detection.

Oh yeah, I just remembered I already created a simple RMS node in the custom nodes example.

https://github.com/BillyDM/Firewheel/blob/main/examples/custom_nodes/src/nodes/rms.rs

Oh actually my implementation doesn't check if the new value is larger than the old one before storing it. I should fix that. 😅

[static quest]

And also technically the correct way is to have the final loudness measure be a sliding window average of the blocks, instead of just using the values directly from the blocks themselves. But again this is probably overkill for games.

(Or at least I think that's how it works. I might need to research how other people have implemented it.)

[short fossil]

I need to know the last 500 ms worth of samples in the ECS so that my AI code can use it. The AI only listens in once every 500 or 200 ms, depending on the distance between the NPC and the player

Their processing is expensive, so I first filter out all sample players that are not loud enough to reach them anyways

And I also downsample the data so that the AI can process them faster

All of this processing happens on the main thread, but I need to get those 500 ms worth of samples from the audio thread

I would be very grateful if you could also do a sanity check, if you happen to have time: https://github.com/janhohenheim/thief_sense_demo/blob/b7c1424f9c48b8d0af9fdea1e5c6eafa8376205c/src/demo/ai/hearing/node.rs

The one thing I know I need to change is the fixed block size

Source for the fixed block stuff is https://github.com/janhohenheim/bevy_steam_audio/blob/main/crates/bevy_steam_audio/src/nodes/mod.rs

but no pressure, I'm already very happy with Corvus having taken a look at it 🙂

[ionic sedge]

Yeah to be clear, Jan is simulating the apparent amplitude of sounds for each NPC. He's running the raw audio through Steam Audio's spatialization processing, which handles things like occlusion and reflection (or at least reflection approximations).

Since it's quite expensive, the audio data is downsampled and injected into the game logic so the processing can be done there.

[short fossil]

@ionic sedge mind giving a listen? 🙂

^ this is the downsampled audio (in no particular order; it's using the shuffle bag)

and these are the orignal audios

[ionic sedge]

ya that sounds correct

[short fossil]

Yay!!!!!

[short fossil]

Looking at the rms, it seems like the downsampled ones are more silent

that's expected, right?

Since the higher frequencies are gone

[ionic sedge]

ya if there's a decent amount of energy in the higher frequencies, then downsampling will reduce the overall amplitude

[short fossil]

is there some rule of thumb factor?

[ionic sedge]

but i don't think it'll affect things much

[ionic sedge]

no it depends on the frequency content of the source

[short fossil]

Definitely not, since NPCs need a custom threshold for alertness anyways

[ionic sedge]

a sine wave at 1khz will be (ignoring any potential precision loss) completely unaffected

as an example

[short fossil]

so I'm pulling some audio multipliers out of my hat anyways

[ionic sedge]

a lot of these footstep sounds have a broad range of frequency content, so part of their energy is lost in the conversion

[short fossil]

is there some physical equivalent to the RMS? like, is it a kind of dB?

[ionic sedge]

only insomuch as the individual sample values have some physical meaning, so not really

but you can convert the amplitude to dB

https://docs.rs/firewheel/latest/firewheel/dsp/volume/fn.amp_to_db.html

[short fossil]

I was just wondering since that would allow me to pick some audibility thresholds that are known from humans

[ionic sedge]

it's all relative, really -- but if you work in dB it should be easier to conceptualize

You'll have to do some tweaking, but you'll want to choose some level in dBFS (what you get from Firewheel) and correlate it to a dB SPL level. Once you've decided where they meet, the relative difference between them is identical (+6dBFS is the same as +6dB SPL).

[short fossil]

Oh, also ChatGPT is telling me something about RMS decreasing by sqrt(resample_ratio)

is that relevant or just hallucinated

[short fossil]

Is dB FS the one pegged to human hearing?

I remember we talked about this when I was wondering about 0 dB not being full loud in my college level physics haha

[short fossil]

makes sense, thanks

[short fossil]

is that the same as doing Volume::Linear to Volume::Decibel?

[ionic sedge]

sorry i hit enter too soon

[short fossil]

looking at the implementation the math looks the same, so is "amp" just a synonym for "linear"?

[ionic sedge]

dB FS (full-scale) is basically just where 0dB is the full scale amplitude (-1 to 1)

dB SPL is expressed in terms of actual sound pressure levels

[short fossil]

Aaaaah this I remember!

[ionic sedge]

so like when someone says something is 80dB, they probably mean 80dB SPL

[ionic sedge]

i think there's some extra processing on the linear variant actually

[short fossil]

okay, so dB FS is what we are talking about all the time when writing about dB in seedling

[ionic sedge]

ya

[short fossil]

does this mean anything to you?

[ionic sedge]

out of context that's incorrect

[short fossil]

yeah that's what I thought

[ionic sedge]

maybe for a broadband signal, like noise

[short fossil]

but thanks for confirming 🙂

[ionic sedge]

but it's frequency dependent like I mentioned -- a sound well within the nyquist frequency of both rates will be completely unaffected

(within any error due to discrete sampling)

[short fossil]

makes sense, thanks

Cool! Now I can finally hand the data over to steam audio haha

I swear I need to add you as a co-author to the Thief AI Demo

I cannot stress how much I would not have been able to do this without you

[short fossil]

Just had a simple idea for preserving energy

I could calculate the RMS of the input chunk

And then scale the output chunk by the ratio of its RMS to the original RMS

which should boost the low frequencies so that the total energy stays constant

Oh actually that would be bad

Since low frequencies get less absorbed by materials, boosting them will make them travel further, making AI audio different from player audio

Nvm I'll leave the energy decrease then

[short fossil]

@ionic sedge if I use OnComplete::Remove, does the processor still exist and process a bunch of 0s as inputs?

[ionic sedge]

which processor?

[short fossil]

A processor belonging to a sampler effect node

[ionic sedge]

even if you despawn the SamplePlayer, the nodes will be completely unaffected.

except that any relationships are severed

[short fossil]

Wait, so does only a single processor exist in that case?

or one per sample player?

(for sample effects)

[ionic sedge]

For each SamplerNode in the pool, there is one corresponding chain of effects. A SamplePlayer can apply new values to any of these nodes, but it doesn't "replace" them in any meaningful way, and when the SamplePlayer is despawned they are simply no longer affected by any state of the player.

[short fossil]

So for

commands.spawn((
    Name::new("AI sound pool"),
    SamplerPool(AiPool),
    sample_effects![InputBufferNode, SteamAudioNode::default(),],
));

and

commands.spawn((
    SamplePlayer::new(foo),
    AiPool,
));
commands.spawn((
    SamplePlayer::new(bar),
    AiPool,
));

How many SamplerNodes are there?

1 or 2

[ionic sedge]

four

[short fossil]

🤯

[ionic sedge]

The default size range for a pool is 4..=32 https://docs.rs/bevy_seedling/latest/bevy_seedling/pool/struct.DefaultPoolSize.html.

[short fossil]

So there are also 4 processors?

[ionic sedge]

yes

[short fossil]

I see!

[ionic sedge]

It's a bit like the length of a vector compared to the capacity, although the unallocated nodes in this case are valid and active.

But like a vector, it's very expensive to create a new allocations (to inject new nodes into the audio graph). So we want to avoid it as much as possible.

[short fossil]

So here

fn establish_channel(
    mut nodes: Query<(Entity, &SampleEffects), Added<SamplePlayer>>,
    mut input_buffers: Query<&mut AudioEvents, With<InputBufferNode>>,
    mut commands: Commands,
) {
    for (entity, effects) in nodes.iter_mut() {
        let Ok(mut events) = input_buffers.get_effect_mut(effects) else {
            continue;
        };
        let (prod, cons) = HeapRb::new(FRAME_SIZE_FAR as usize).split();
        let event = InputBufferInitEvent(Some(prod));
        events.push(NodeEventType::custom(event));
        commands.entity(entity).insert(InputBuffer {
            inputs: VecDeque::from_iter([0.0; FRAME_SIZE_FAR as usize]),
            loudness: 0.0,
            cons: Mutex::new(cons),
        });
    }
}

Which processor is getting these events?

[ionic sedge]

The one that is eventually assigned to the SamplePlayer.

[short fossil]

Oh so the events are buffered until a processor is available?

[ionic sedge]

ya

[short fossil]

Okay got it, this helped a lot

[static quest]

Ok, but, why not just poll the RMS value every 500/200 ms? Computing RMS is very cheap with the method I described.

Or do you mean that you want to re-run the Steam Audio's spatialization algorithm multiple times for every AI observeror? Hmm, I guess that is one way to do it.

Though if it were me I would just cast a few rays between the sound source and the AI observors in the game loop. Then take the raw RMS value of the sound source, and decrease the RMS value according to the average distance of each ray (volume decreases by the square of the distance). And if a ray goes through a wall, just increase the "distance" of that ray. Of course this wouldn't be as accurate (especially if you want the AI to be able to tell the difference between high and low frequencies), but if all you want is to have AI react to the volume of sounds, this would be way cheaper than running the entire sound spatialization algorithm for each observor.

[short fossil]

I need to run Steam Audio every 500/200 ms, yes

For most games that approach you describe would be enough, yes. But I'm specifically recreating Thief's AI, which depends on being able to hear the player accurately

[static quest]

Hmm, ok.

[short fossil]

The Thief devs made their own audio spatialization pipeline for the AI, so I'm happy I can at least just reuse Steam Audio haha

(the player audio was spatialized through hardware acceleration using sound cards)

[static quest]

Oh, crazy.

[short fossil]

Yeah Thief's tech stack is unique. And that in 1997!

[static quest]

Still, I'm a bit skeptical of how much better doing a full audio spatialization would actually be over just casting rays for the RMS value. At the end of the day, sound pressure is just sound pressure, and it is practically a linear equation based on the distance.

[short fossil]

I also need the direction where the sound came from

So the NPCs go look there

[static quest]

And that can also be solved by casting rays in the game loop.

[short fossil]

Yeah, but at that point I'm basically recreating steam audio lol

[short fossil]

I'm using their direct effect (= ray cast) and pathing (= 3D pathfinding)

I already have steam audio in the app for the player sound output anyways, so I can reuse the Steam Audio scene (= BVH) and audio sources

[static quest]

Well, not exactly. You'll only be recreating the raycasting part, not the processing every 48000x2 samples per second part.

[obsidian tusk]

There’s bindings for Steam Audio called AudioNimbus, Steam Audio has incredibly powerful spatialisation/occlusion/etc

[obsidian tusk]

Oh sorry 😅 Joined late

[short fossil]

Corvus and I made a library for integrating audionimbus with Bevy! https://github.com/janhohenheim/bevy_steam_audio

[obsidian tusk]

👀👀👀 That’s kinda exactly what I was looking for! I was going to have a crack at it myself actually

[static quest]

My point is that because sound pressure is pretty much linear, you will get pretty much the same result if you do the RMS first and then the raycasting as you would if you did the raycasting and then the RMS.

[short fossil]

Fair, I need to process the samples. But I'm just decimating them to 8k, which should be plenty fast

[short fossil]

Ooooh I see

Yeah, good point

But uuh I am already doing that

I do the RMS before doing anything fancy

[static quest]

Hmm, I guess. (But at a sampling rate of 8k that means the AI won't be able to hear frequencies above 4k).

[short fossil]

yep. I think that should be fine, but I'll have to bump that up if not

[static quest]

Well, except for maybe reverberation and echos.

[short fossil]

Yeah, and I'm not simulating those anyways

So yup, you're absolutely right in that that would be a valid approach

I just think I'm better off letting Steam Audio handle it instead of reinventing the wheel there, at the cost of having to do the firewheel -> downsample buffer -> ECS song and dance

But do mind that there's maximum 10 sources in the level that the AI cares about

Usually just 2-3

[static quest]

Ok

[short fossil]

You don't sound convinced 😅

Does it make more sense when I say that I need to factor in which path the sound took to get to the NPC?

[static quest]

I mean, that's just raycasting?

I don't know, it just seems like code smell to me. But you do you.

[static quest]

If you are casting rays, you also know the angle of the rays.

[short fossil]

no please, I may be missing something obvious

[short fossil]

I'm currently doing a 3D pathfinding to approximate the way in which the sound bounced around corridors to reach the NPC

I'm not sure how I would do that with raycasts?

Except doing a full reflection simulation of course, raycasting somewhere and then bouncing by raycasting again recursively

But you probably don't mean that, right?

[static quest]

Oh, ok. 3D pathfinding works for that too. (In fact you could probably just approximate everything by using the distance of this path as the "average distance of the rays", without having to do any raycasting at all.

(Though maybe you might want to also factor in how many walls there are directly between the sound and the observor.)

I guess I'm just thinking of this in terms of "what do we really gain by trading simplicity for accuracy"? Does it really matter that the AI gets the exact RMS value, or is it good enough that it just gets a rough idea if the volume is greater than a certain threshold?

[short fossil]

yeah rough idea is definitely fine. It currently works like this:

• Calculate the RMS and do trivial distance falloff to see if the NPC has any chance of hearing the sound at all. No raycasts.
• If that worked:
  • do up to n raycasts (I think I set 4 rn) to get the direct sound to the NPC (factoring in walls on the way)
  • do 3D pathfinding to get
    • the actual RMS
    • the path that the sound traveled through so that the NPC can backtrace it, depending on various factors

And since I already have Steam Audio set up for the player, most of this can reuse existing stuff, like the scene for pathfinding

That's why to me it feels simple, from a code perspective

Does it make more sense now? 😄

[static quest]

Ah yeah, I can see why reusing the collision for Steam Audio would be helpful.

[ionic sedge]

You are simulating the pathing, right? That's an approximation of reflections, I believe. It simulates the shortest, non-occluded path from source to listener. This gives the simulation much more information than a mere raycast, especially when your level is lots of tight corridors.

[short fossil]

Yep, precisely

[obsidian tusk]

Maybe you could prebake the info, if your levels are static? It wouldn’t be much different to lightvol calculation. Quake 3 has potentially audible set calculations that I think take multiple bounces into account (but don’t store attenuation etc)

Like, for an example of prior art

[ionic sedge]

pathing
im a ninja

[obsidian tusk]

Then you can use an equivalent of area portals to handle opening/closing doors

[ionic sedge]

(pathing uses baked probes)

[short fossil]

The pathing probes are prebaked for the level with all doors and shortcuts opened, yeah. Steam Audio then verifies the path by doing raycasts. If they fail, it searches alternatives

[short fossil]

I could supplant this with actual vis portals, but my level design time takes ages already haha

[obsidian tusk]

Yeah area portals are super time consuming, especially if it affects gameplay and isn’t just an optimisation

[short fossil]

Do I‘m exploring ways where it Just Works with any level geometry without any level designed input

In my experience, the bottleneck always ends up being the Bevy renderer anyways

[obsidian tusk]

Yeah I think that’s sensible for a solo dev

[short fossil]

I have some plans for how to deal with that once it becomes too annoying

(Custom PBR material and some funky heuristics to disable dynamic lights that are out of view)

But I'll burn that bridge when I get there

But yeah in my experience the biggest time bottleneck is level design and designing props

So I'll gladly trade performance for that haha

@ionic sedge is AudioNodeProcessor::new_stream called when a new sample player gets a spot in the pool?

Asking because I just learned that we're supposed to call effect.reset() when a new source starts playing

[ionic sedge]

only when the entire stream is restarted

[ionic sedge]

i’d add a Notify<()> field for this

[short fossil]

In the Node?

[ionic sedge]

ya the steam audio node

[short fossil]

Should node configs be Diff/Patch?

Or only the node itself?

(I assume the latter)

[ionic sedge]

ya we dont use diff and patch for configs

[short fossil]

Dunno if you knew this, but

I'll #[reflect(ignore)] for now

[ionic sedge]

do you have reflect enabled for firewheel?

[short fossil]

I would expect bevy_seedling to enable that transitively, no?

[ionic sedge]

ya

[short fossil]

Oh, strange

Activating it on firewheel explicitly fixes it

oh I know why

bevy_seedling = { version = "0.6.0", default-features = false }

[static quest]

Oh actually, this crate might be useful to you. 😁 https://crates.io/crates/fixed-resample

[short fossil]

I saw that!

I tried using it but it was too hard for my non-DSP head :/

But yes it looks perfect for what I wanted

So I took the relevant parts that I understood hahah

that's why it uses ringbuf directly

[static quest]

Ah ok, yeah. Essentially it's rubato and ringbuf put together (with some more complicated stuff for dealing with jitter).

[short fossil]

yeah you'll find that the code I posted will look very familiar haha

The jitter parts I left out

[static quest]

Although you might still need to worry about jitter in your case (such as handling the case where the user's CPU can't keep up with the audio stream.)

[short fossil]

Makes sense

I'm probably going to try integrating it again later

may I ping you when I get stuck? 😄

[static quest]

Yeah, the simplest solution is to just discard samples if the buffer gets too full.

[short fossil]

I think that's what my implementation does now?

Not sure

I haven't thought it through

[static quest]

Hmm, though the clicking noise caused by discarding samples might be a bit problematic in your use case. (Though you downsampling to a smaller sampling ratio would make the clicking noise less prevalent.)

At some point I've been meaning to add declicking to fixed-resample. It just, complex and I didn't feel like bothering with it yet.

Streaming audio in general is just notirously difficult to get right.

[short fossil]

discarding stuff causes clicking?

(again I'm very new to all of this haha)

[static quest]

Yes. Clicking occurs when a signal abruptly changes.

[short fossil]

ooh gotcha

BTW I don't know if I already told you this, but working with the whole firewheel setup is lovely

Y'all did a really good job

[static quest]

Hmm, then again, I don't think clicking contributes much to RMS.

[short fossil]

I love how seemingly for every problem I encounter there is a solution 🙂

[static quest]

So you might be able to get away with it.

Yeah, now that I think about it, clicking would probably barely even register in the RMS calculation. (Though if you had reverb and echos, then it would.)

[short fossil]

good thing I don't have those

I'll quickly finish the effect reset stuff and then try the fixed-resample crate again

@ionic sedge when should I notify? i.e. is there some event I can watch to know when a new sample player has slotted in?

[ionic sedge]

Sampler added to entity with SamplePlayer which has it as an effect

[static quest]

Thank you! I can't wait to see what people do with it.

[short fossil]

Power AI apparently 😛

No for real, it's great for the player audio

I was also looking a little bit into FMOD, and correct me if I'm wrong, but it seems like seedling should be able to do most things that FMOD can?

[ionic sedge]

in theory 😅 but it would be a lotta work

[short fossil]

(I have never used FMOD, just heard that everyone and their dog uses it)

[static quest]

Yeah. The goal of Firewheel is to be a competent alternative to FMOD/WWise. (Though without the fancy editors and stuff.)

[ionic sedge]

but id love to work towards it! an editor will be a real step change in usability and artist-friendliness

for bevy

[short fossil]

And a step up in "Corvus explains to Jan how a graph works" technology

[static quest]

I do still need to revamp the sampling engine to be able to handle looping. But I have other priorities at the moment (namely getting my DAW off of the ground).

Firewheel has been a great learning experience. I've essentially been rewriting my entire DAW engine based on what I learned. 😅

[short fossil]

Like this?

fn reset_reverb_node(
    add: On<Add, Sampler>,
    effects: Query<&SampleEffects, Allow<Disabled>>,
    mut reverb_node: Query<&mut SteamAudioReverbNode>,
) -> Result {
    let effects = effects.get(add.entity)?;
    let Ok(mut node) = reverb_node.get_effect_mut(effects) else {
        return Ok(());
    };
    node.reset.notify();
    Ok(())
}

[short fossil]

I also want to get into DAWs at some point. Would you say your DAW would be adequate to learn the basics?

[static quest]

As in learning how to make music/sound effects using a DAW?

[short fossil]

yep

Well, or basically just learning a bit more about sound production

no need for "music" to come out haha

[static quest]

FL Studio is still king when it comes to user-friendliness. But of course I'm striving to make my DAW user friendly as well.

(And also just the sheer amount of factory content included in FL Studio)

Presonus Studio One is also a great beginner DAW.

And in terms of synthesizers, Vital is incredible for being a free synthesizer (and it's also FOSS!)

[short fossil]

never heard of it, thanks for the rec 🙂

[static quest]

A lot of the DSP for my DAW will actually be based on the DSP of Vital.

(I've also already ported Vital's reverb module to Rust. https://github.com/BillyDM/vitalium-verb)

[short fossil]

I really want to also get into https://github.com/j-p-higgins/SoundThread / CDP, I love the fucked up sounds I saw people make with it haha

[static quest]

Oh yeah, I've heard good things about that.

You might also be interested in Bespoke Synth if you like the node-based workflow. https://github.com/BespokeSynth/BespokeSynth

[short fossil]

Oh wow that looks neat 👀

Alright fixed the reset issue

now let's get to the resampling

@static quest the docs tell me to use NonRtResampler

but that seems to not exist?

Or should I rather look at the CPAL loopback example?

yeah that one has the (prod, cons)

[static quest]

Oh whoops, that's outdated. I used to have both a realtime and a non-realtime variant, but I since figured out a way to combine them into one.

[short fossil]

Do I call fixed_resample::resampling_channel from inside a node or in the ECS?

Since it's asking be for the input sampling rate, which AFAIK I don't have in the ECS

[static quest]

Yeah, it will have to be when you construct the node.

[short fossil]

By "construct the node" do you mean construct the processor?

[static quest]

Yeah, sorry that's what I meant

[short fossil]

dw Corvus told me y'all use "node" interchangeably for the node and processor haha

[static quest]

Yeah, lol. The processor is still a "node", it's just a node in the processing graph instead of the graph on the game thread.

Really the only reason the API makes a distinction between the node in the main graph and its processor is it to satisfy Rust's thread safety requirements. In C/C++ land we could have just designed it to where they are both the same struct. (In fact this is how audio plugins like CLAP/VST work.)

[short fossil]

How come your reader node doesn't create the resampling channel in construct_processor, but instead want the prod over an event?

referencing this: https://docs.rs/firewheel-nodes/latest/src/firewheel_nodes/stream/reader.rs.html#108-136

[static quest]

Oh right, i did design it that way.

[short fossil]

In my current code, I have this neat setup where I create the actual resampler in construct_processor (where it knows the sample rates), and the (prod, cons) pair in the ECS

[static quest]

Actually, in fact I think you are able to just clone StreamReaderState and send that to whatever thread you want.

[short fossil]

What's StreamReaderState?

[static quest]

Oh yeah, that's how I did in in the example. https://github.com/BillyDM/Firewheel/blob/3b2af9fe3282edf53c15b57d2f3f673ca705cb27/examples/stream_nodes/src/main.rs#L115

[short fossil]

Idk what the analogous setup for me is :/

[static quest]

Admittedly the API I used probably isn't the most friendly with Bevy.

[short fossil]

like, do I also need a SharedState and an ActiveState

[static quest]

No, that's only if you want users to be able to read the samples from the game thread.

(Though you will probably need a shared state so that users can read the final RMS values)

[short fossil]

so do I still create the (prod, cons) pair in construct_processor?

[static quest]

So I think in your case, the easiest solution is to create an Arc<Mutex<Option<ResamplingCons>>>that you store in both the node's constructor and whatever thread you do your calculations on. Then just populate the consumer when the node processor is constructed in construct_processor.

[short fossil]

okay that sounds actionable 🙂

[static quest]

(Or use a channel to send the consumer to the thread you want.)

[short fossil]

a channel to send the channel

a channel-channel

[static quest]

Yeah, lol. That's realtime programming baby!!! 😎

Audio programming in general would be much simpler if Mutex was realtime safe. (In fact some game engines like Godot don't even bother and just use mutexes anyway.)

The thing about Mutex is that the vast majority of the time it doesn't cause issues, but occasionally you might get hard to explain skips in your audio.

[short fossil]

yeah I remember you said it allowed the OS to park

[short fossil]

I assume by "the node's constructor" you mean the thing that implements AudioNode?

[static quest]

Yes

[short fossil]

And it doesn't need to be an ArcGc right?

[static quest]

(Sorry for using confusing terminoligy again, lol)

[short fossil]

Since it's never even touched on the audio thread

[static quest]

Yeah, it doesn't need to be ArcGc if it doesn't touch the audio thread.

[short fossil]

@static quest alright I managed to wrangle the cons, prod pair

now I need to push and read, right?

since I only have one channel, I can use the _interleaved() variants, right?

[static quest]

Yeah, that should work.

[short fossil]

for the read, what size does the buffer need to be?

and do you have some guidance on what the config should be, other than using low quality?

[static quest]

It's designed to work with any sized buffer.

[static quest]

I would set latency_seconds to be just a bit larger than the size of your buffers.

So, say, if you wanted to process 500ms worth of audio at a time, set it to something like 550ms.

capacity_seconds should be at least twice the length of latency_seconds (though it automatically sets the capacity to be at minimum twice latency_seconds).

And in your case I think it makes sense to set underflow_autocorrect_percent_threshold to None.

[short fossil]

fn resampling_config() -> ResamplingChannelConfig {
    ResamplingChannelConfig {
        quality: ResampleQuality::Low,
        capacity_seconds: 2.0 * SENSE_INTERVAL_FAR as f64 + 0.1,
        latency_seconds: SENSE_INTERVAL_FAR as f64 + 0.05,
        underflow_autocorrect_percent_threshold: None,
        ..default()
    }
}

thx

[short fossil]

So can I safely size it to my fixed block size, as an arbitrary value?

[static quest]

Yeah, what it does internally is fill the buffer you give it as much as it can (and if there are no more samples left in the channel, it fills the rest with zeros).

[short fossil]

is this correct then?

let mut scratch = [0.0; FIXED_BLOCK_SIZE as usize];
let status = cons.read_interleaved(&mut scratch);
match status {
    fixed_resample::ReadStatus::Ok => FIXED_BLOCK_SIZE,
    fixed_resample::ReadStatus::InputNotReady => 0,
    fixed_resample::ReadStatus::UnderflowOccurred { num_frames_read } => {
        num_frames_read
    }
    fixed_resample::ReadStatus::OverflowCorrected {
        num_frames_discarded: _,
    } => FIXED_BLOCK_SIZE,
}

[static quest]

Yep! (And you probably don't even need to worry about the status. It's only there if you need to do something special if the input overflowed/underflowed.)

[short fossil]

neat!

[static quest]

Though you might want to use it for debugging to dial in the latency/capacity.

(If you get too many underflows, increase latency. If you get too many overflows, increase capacity.)

[short fossil]

I see lol

Oh I forgot to mention: I'm updating the input buffer every frame

[static quest]

Also make sure to test it in release mode. Sometimes the internal resampler can be unusually slow in debug mode.

[short fossil]

it's a sliding window of the last 500 ms

[short fossil]

I already have optimizations for all deps on 🙂

This is giving me an underflow all the time

wait, if my latency is 1 sec, wouldn't that mean that NPCs listen 1 second into the past?

ah no I set it to 550 ms

so yeah makes sense that they listen that far into the past

[static quest]

Oh, yeah, reading it every frame will definitely cause a problem. (1/60 = 16ms, much smaller than 500ms).

[short fossil]

Yeah I use a sliding window because each NPC reads the state staggered

[ionic sedge]

But you don't need all 500ms at once if you're maintaining a sliding window in the ECS.

You can just pull off whatever samples are available, no?

[short fossil]

yep that makes more sense

[static quest]

Ah, the resampling channel is streaming, meaning that once you read from it, all of those samples are popped from the buffer.

So yeah, in your case it might be better to set the latency to something like 30ms and only read 16ms worth of data from the buffer each frame.

Sorry, didn't know you were using a sliding window. Reading 500ms worth of data from the buffer each frame will definitely not work for that.

[short fossil]

I'm running this in a fixed timestep, so my dt is always 16.6 btw

[static quest]

Oh yeah, and keep in mind that the audio thread itself updates anywhere from around 10ms to 100ms depending on the block size. So in your case it might make sense to leave the latency at the default value of 150ms.

[short fossil]

and leave underflow_autocorrect_percent_threshold on None?

[static quest]

Though you could probably get away with around 100ms. I chose 150 as the default just to be safe.

[static quest]

Correct. That's only useful if the consumer is used in a realtime thread (I should probably make that a bit clearer in the docs.)

Honestly the fixed-resample crate needs better docs and examples in general. It's just a tough concept to explain to users.

[short fossil]

yeah I believe that haha

[static quest]

And maybe a higher level API for the more common use cases.

But of course that would be more work, and I'd rather work on other stuff at the moment. 😅

[short fossil]

so just experimentally, I get these values

Based on that, how big should by buffer be?

[static quest]

And even then, it would make sense that an NPC would need some time to react to a sound.

[short fossil]

at 8k Hz, 1/60 of a second should be 134 samples if I'm not mistaken

[ionic sedge]

keep in mind that the important thing here is the wall-clock rate between the producer and consumer -- the fixed time step may run multiple times per frame, or not even once, and if it goes too long between reads the buffer may overflow

[short fossil]

yeah definitely

[short fossil]

oh right I didn't consider that

[static quest]

Yes, that is correct. You would want to read 134 samples at a time at 60fps.

[short fossil]

that gives me this

[static quest]

Yeah, that's what I meant by "jitter". fixed-resample automatically handles jitter for you!

[short fossil]

is that acceptable?

[short fossil]

Makes sense. So I should remove it from the fixed timestep

[static quest]

Hmm, shouldn't be getting so many "Read 0 frames".

[short fossil]

multiple fixed timesteps in one frame

Though wait, I shouldn't get 4 of those

The reason why I grabbed a fixed timestep at all is because I'm backfilling my ECS buffer with 0s if the sample stopped playing, so that NPCs have a chance to hear samples that were too short

And for that I would have used Res<Time>, which is often an indicator that you should be using a fixed timestep for it to stay deterministic

but I see that having this be deterministic is a fools game lol

[static quest]

So again, it is streaming, meaning every time you read from the channel, it pops that many number of samples. So reading multiple times in a single frame won't work.

[short fossil]

got it

this is now running it every frame

[ionic sedge]

What's your frame rate?

[short fossil]

oh

you're right, this is no longer a fixed timestep, so it's 165

haha

[static quest]

Oh, actually, nevermind. Reading this closer I didn't realize you were talking about frame pacing on the game engine side. Nevermind.

[short fossil]

Alright, then back to the fixed timestep!

[ionic sedge]

I don't really see how it matters tbh

You should expect to read 0 samples if the frame rate doesn't line up perfectly with the audio processing rate.

[short fossil]

You're saying the underruns are fine, I presume?

[ionic sedge]

ya

the sliding window is not sensitive to them

[short fossil]

Cool beans

[ionic sedge]

i mean unless the audio graph itself actually underruns

but you can't really do much about that

[short fossil]

Makes sense to me, I just didn't know if the resampler would somehow not like that

[ionic sedge]

Ah, well maybe there's something going on there that I'm not aware of.

But if the resampling is happening in the node, it should be okay.

[static quest]

Actually, no, that's what the latency is for. To hold on to a buffer of samples so that you never run out even if the timings aren't perfect.

I think the issue was that he was reading from the channel multiple times per video game frame, instead of reading at a fixed 60 times a second.

[ionic sedge]

Oh I see the issue.

[short fossil]

I can check, sec

[ionic sedge]

Well yeah the fixed time step would ensure you always pull the correct amount out.

Otherwise you'd have to check frame deltas.

[static quest]

Because if you weren't using a fixed timestamp, you would constantly have to change the size of the buffer you are reading to match the amount of time between the current frame and the last one.

[ionic sedge]

Either way, the underruns don't really matter. In fact, I'd pull out the samples as eagerly as possible and use a latency of zero. For this use case, a steady stream doesn't matter.

[static quest]

And you can use frame delta, though I can imagine you can get cumulative errors over time due to rounding.

[ionic sedge]

(Unless I'm missing something, but the sliding window should mean that it doesn't matter.)

[static quest]

But they do matter? If you have too many underruns, then you will have too many zeros in your signal, and the resulting measured RMS would be much lower than it should be.

[ionic sedge]

If samples aren't available then you don't push to the sliding window.

[short fossil]

I'm definitely not doing that, no

(I double checked with debug printing)

[static quest]

Oh, I get what you are saying.

[short fossil]

Yeah I'm doing this:

if incoming == 0 {
    // be kind to change detection
    continue;
}
buffer.inputs.drain(..incoming);
buffer.inputs.extend(&scratch[..incoming]);
buffer.update_loudness();

[ionic sedge]

If you want a nice steady stream, then a tiny bit of latency and avoiding underruns is definitely correct. But I think it might be better to have the minimum latency for the AI and I don't think they'd be sensitive to little hiccups.

In other words, if no new data is available from the audio stream, then they'll just use whatever is in the window, which should be totally fine for the purposes of AI systems.

[static quest]

Ah ok, yeah, that should work then. I was just assuming that you were using the values in the scratch buffer directly in the RMS calculation. Sorry about that. So yeah, you don't even need latency at all if you just wish to push whatever is there.

[ionic sedge]

But it's not a big deal either way.

[short fossil]

so in that case fixed-resampling isn't doing anything for me, right?

[static quest]

Correct. Sorry about that. 😅

[short fossil]

haha all good

I learned from it 🙂

[ionic sedge]

Sorry 😅 I probably could have helped clarify a little too!

[short fossil]

It's cool. I really do learn a ton about DSP just from troubleshooting with you two

[static quest]

Although, actually it still might be helpful for correcting for overflows (when the AI thread is too slow to keep up with the incoming audio data).

But that's as simple as just clearing the ring buffer if it reaches a certain capacity.

[short fossil]

oh right I didn't consider that, thx

in other news: here's the progress

the red bar is the NPC's RMS perception of the player footsteps, without any Steam Audio yet

so just raw sound, no distance falloff or anything

but it seems to work!

[static quest]

@lapis stone Oh yeah, sorry about being slow on the PRs. Do you feel the convolution node is ready to merge?

[short fossil]

@ionic sedge should we also clear the fixed block when the sample player changes?

[static quest]

Once that is merged, I might go ahead an publish another release of Firewheel.

Oh right, @ionic sedge were there any changes you wanted to make to the bevy derive stuff?

[ionic sedge]

did we ever fix the false cycle detection for zero in zero out nodes?

I’ll make an issue for it if it’s not on your radar

[ionic sedge]

ah, yes i started on some feedback, but i didn’t finish the reflection part (just components)

ill also make an issue there if necessary

[static quest]

Oh, was that an issue? I wasn't aware of that. 😅

[short fossil]

I think so 👀 since otherwise we're accumulating inputs from the last sample, no?

[ionic sedge]

Yeah I’ll get a repro going soon. I’ve also been stalled a bit lately (super busy at work) but I’d like to get things tidy for 0.17.

[short fossil]

@ionic sedge I mean for the fixed blocks in general, not even for a specific node

[static quest]

Oh yeah, @ionic sedge the convolution node adds a small breaking change to SampleResource FYI.

[ionic sedge]

oh is it just in the interface? i’m probably insulated since i don’t think i interact with it directly (except to create it with symphonium)

[ionic sedge]

in the general case no i don’t think so

it’s a little tricky how it interacts with silence (i might need to make it a bit more robust tbh )

[static quest]

Alright, fixed!

(I simply forgot to count the zero in zero out nodes in the compiler)

[hard plaza]

hi, i'm trying to get input from a mic with bevy_seedling, but it causes my program to panic immediately after starting the output stream
am i doing something wrong?

this is the log with RUST_BACKTRACE=1

and my plugin config:

SeedlingPlugin {
    stream_config: CpalConfig {
        input: Some(CpalInputConfig::default()),
        ..default()
    },
    ..default()
}

[static quest]

Hmm, I must have done something wrong here. I'll look into it! https://github.com/BillyDM/Firewheel/blob/3b143eb0b705692bcc76cf1f0e3c4465bd7061cf/crates/firewheel-cpal/src/lib.rs#L884

[hard plaza]

alright, thanks for looking into it 🫡

[static quest]

Ok, I pushed a fix to the main branch that should fix the panic. (I forgot that some platforms may not actually use a fixed block size when you request one.)

@ionic sedge Should I push that fix to cargo.toml, or should we just wait until we get the next release of Firewheel ready?

[ionic sedge]

oh as in crates.io? hm, i think we should be able to publish everything in the next couple days, so no rush!

[short fossil]

Ping me when you do, then we can also publish bevy_steam_audio 🙂

[static quest]

Oh derp, yeah I meant crates.io. (I'm tired today 😅 )

It would be funny if cargo.toml was a valid url

[lapis stone]

i think it should be good!

[hard plaza]

thanks!

[hard plaza]

yep, just tested, can confirm that panic is fixed now

[short fossil]

@ionic sedge I added you to bevy_steam_audio's authors: https://github.com/janhohenheim/bevy_steam_audio/pull/44

Also added you to the funding file

Which you may want to also set up in seedling 😉

[ionic sedge]

Oh cool, thanks!

[ionic sedge]

no i must destroy bevy_seedling
it must be assimilated

[short fossil]

https://github.com/CorvusPrudens/bevy_seedling/pull/68

welp here is the PR anyways 😛

there we go

beautiful

Also sent you some thank you money

[ionic sedge]

aw that's so nice

[static quest]

Funding file you say? 😉

Though I am a bit hesitant on whether I should keep my GitHub Sponsors profile or not (I'm trying to move away from GitHub wherever I can.)

[static quest]

Alright, merged!

@ionic sedge Oh yeah, do you think I should add a quick highpass filter parameter to the wet output of the freeverb & convolution nodes? (For removing low-end rumble.)

[ionic sedge]

ya probably
also I think it's set up now to only produce wet output

That API came from the original code, but I believe it's unused in the node itself.

imo that probably makes sense -- we may want a more unified, graph-based way to manage dry/wet

[static quest]

Oh yeah, you're right, there is no mix parameter on the freeverb node.

[ionic sedge]

as compared to a bit more ad hoc settings on some effects

[static quest]

In that case users can simply add a highpass filter node after the reverb node if they want to remove low-end rumble.

[ionic sedge]

well that is true!

[static quest]

I wonder if the mix parameter should be removed from the convolution node as well?

Though actually, convolution can be used for more than just reverb, so it probably makes sense to keep it.

[short fossil]

I think if you're moving off GitHub, there's nothing wrong with keeping GitHub as a paying method for others to pay you. But yeah I definitely get the impulse to de-GitHubify

[short fossil]

hehe I see

[static quest]

Fair. Although I do also have a liberapay and a BuyMeACoffee.

[short fossil]

Conversely, I should really set those up at some point 👀

[static quest]

But I suppose GitHub sponsors might be a bit more friendly for users/companies that already use GitHub.

I also need to set up a Patreon for Meadowlark at some point.

(Also, fun fact GitHub Sponsors, Liberapay, and BuyMeACoffee all use Stripe as their backend, so they are all kind of the same thing in the end.)

Although I think Liberapay also has a PayPal option (or was it BuyMeACoffee?)

[short fossil]

ihatestripeihatestripeihatestripe

[static quest]

Yeah, it's unfortunate.

[short fossil]

still mad about the whole NSFW game purge thing due to payment processors

[static quest]

I went ahead and added a FastRmsNode to firewheel_nodes! (I'm calling it "fast" because it doesn't calculate the true RMS value which requires a much more expensive sliding window algorithm, but it should be good enough for games that simply wish to react to player audio.)

[hard plaza]

SpatialListener3D docs say

Multiple listeners are supported. bevy_seedling will simply select the closest listener for distance calculations.
Does this mean listeners other than the closest one to the audio source don't output any audio?

[obsidian tusk]

I don’t think listeners output audio at all, they’re just used to attenuate the actual source

Although I have suggested before that listeners should essentially be mixers

The current system is a good default since it handles (for example) split-screen without phasing, but it’d be nice to have some configuration

Do you have a specific goal for having multiple listeners active on the same source? The only one I can think of off the top of my head is for having something like an in-universe microphone/speaker arrangement

[hard plaza]

basically what you described, yeah

in-universe mic/speakers

[obsidian tusk]

I know that in the Source engine they support microphones but will choose only one of the player or the microphone to be the listener and apply effects accordingly, you need to design your levels around it so it’s not too noticeable but if that works for you it’s a fairly reasonable system IMO and pretty efficient

Like, they choose one based on distance (just like in seedling)

[ionic sedge]

Can you expand in this more?

[obsidian tusk]

Ok so I had a think and I think I’ve got a solution. Basically you duplicate bevy_seedling's Send effect into a new effect. In the new effect's config you have some kind of "listener_entity" field (this will be useful in a sec) as well as duplicates of all the fields in SpatialBasicNode. In the new node's AudioNodeProcessor, you call out to the process method of SpatialBasicNode's AudioNodeProcessor, but split the resultant audio like the Send effect does so you can pass it to both the main bus and the microphone's bus. For the microphone, you have your own marker component (e.g MicrophoneListener3d), as well as making it an audio bus, and you have a system to calculate the offsets between the microphone and the entities that have your custom Send applied to them, setting the offset field accordingly. Add whatever fx you want to the microphone, route it to another "speaker" bus that has the regular seedling SpatialBasicNode applied and the relevant transform, and then route the speaker node to your main bus

Since you're here now corvus, can you tell me if that’s a good idea or if I’m giving bad advice? 😅

[ionic sedge]

It’s a touch annoying because the sampler pools abstraction wants a single output for all samplers. The capability is certainly there in bevy_seedling (and Firewheel) to support true multi-listener processing. But it would be awkward with the current sampler pool abstraction.

[hard plaza]

just as an example, a CCTV system
a camera entity picks up sound with a listener, and an emitter somewhere else plays what the listener is hearing, which the listener on the player character will then hear

[ionic sedge]

I think your suggested approach @obsidian tusk might not be totally complete. But I can’t fully evaluate it atm.

You’d basically want two (or however many listeners there are) separate processing chains for each sample player.

[obsidian tusk]

Does the sampler pool support sends, or am I misremembering that? Could this be addressed in seedling by adding some kind of way of adding an effect to the send node (or just having a new SpatialSendNode) as well as having a way of filtering listeners in SpatialBasicNode and/or SpatialSendNode

[ionic sedge]

A send is just a node, so ya you can slot it in there.

[obsidian tusk]

Would it be possible to allow sends to have arbitrary chains applied, with a similar pooling system to sample pools? I guess if you implemented that it’d probably make more sense to have the sample pools handle pooling the chains for any send nodes too rather than having a separate type of pool

[ionic sedge]

You could calculate this as a single set of spatial parameters to apply to the sampler.

[ionic sedge]

yeah you could build an abstraction like this, probably with a special kind of node

[obsidian tusk]

I think usually you'd want to bitcrush, distort and/or filter the audio that gets sent to the microphone, that’s the usecase I had in mind at least

[ionic sedge]

ya but in the stated case at least, that could also he applied once

it would only be a requirement if the user could hear both streams at once

[hard plaza]

not sure i understand

[obsidian tusk]

Yeah that’s basically what I originally proposed too 😅 You'd still need to use sends though, setting send volume to zero or main volume to zero depending on whether the mic is active, since otherwise you need to rebuild the graph when a sample switches between one or the other

The arbitrary fx on chains is just to put the SpatialBasicNode on the send path

I think a custom node that combines the send and spatial nodes is probably the solution I’d go with

[hard plaza]

sampler being an object playing a sound?

[ionic sedge]

I’m on mobile so I can’t probe enough at the problem atm. But there’s lots of options, including fully processing all listeners and emitters without any tricks.

I can get back to this later.

[hard plaza]

basically, just being able to get audio streams from listeners that can then be processed/rerouted/etc would be ideal

[obsidian tusk]

My laptop is updating right now but if I get time later I’ll have a crack at implementing my idea, I’ve been doing a lot of non-audio work recently and I wouldn’t mind a small audio project to work on

[obsidian tusk]

You can already do that using buses, it’s just that handling either multiple listeners or routing audio to different buses based on which listener is active takes a bit of work

[hard plaza]

oh yeah another concrete use case would be in a VR game, where one listener is for the player in the HMD and one listener elsewhere goes to the regular desktop audio

[obsidian tusk]

Like, the listener isn’t the one producing the audio, but you can still reroute the audio from the sample players based on the active listener, probably using sends and setting either send or main volume to zero rather than actually rerouting since then you don’t need to recalculate the graph

[ionic sedge]

This is 100% something you can’t approximate.

[sharp ridge]

Hi. When are you planing to make it public? Is it still in early dev or usable? 🙂

[static quest]

It's still a work in progress. (It has gone through many rewrites, including a final rewrite using everything I learned from creating Firewheel.)

It's hard to say exactly when it will be usable. I'm aiming for by the end of this year, but I can't make any hard promises.

[ionic sedge]

Okay, so if you want to truly process audio like this with no tricks or shortcuts, then you may want to build a little abstraction on top of bevy_seedling. This is because it clashes a bit with the assumptions made for the sampler pools.

A normal sampler pool in bevy_seedling looks something like this:

┌───────────────┐┌───────────────┐
│SamplerNode (A)││SamplerNode (B)│
└┬──────────────┘└┬──────────────┘
┌▽──────────────┐┌▽──────────────┐
│SpatialNode (A)││SpatialNode (B)│
└┬──────────────┘└┬──────────────┘
┌▽────────────────▽┐              
│VolumeNode        │              
└┬─────────────────┘       
┌▽─────────────────┐              
│Output            │              
└──────────────────┘         

This pool has two samplers, each with a single chain of processing that feed into a bus (in this case the VolumeNode). Each spatial node assumes a single primary listener. Each node is processed with this spatial configuration, and the results are mixed together in the bus.

This doesn't work in a true multi-listener configuration. Each listener needs a separate spatial effect applied at the minimum. If you want to do additional processing afterwards as well, the whole abstraction falls apart.

Really, what you want is something like this:

┌───────────────────┐┌──────────────────────────────────┐               
│SamplerNode (A)    ││SamplerNode (B)                   │               
└┬─────────────────┬┘└───────────────┬─────────────────┬┘               
┌▽───────────────┐┌▽───────────────┐┌▽───────────────┐┌▽───────────────┐
│SpatialNode (A2)││SpatialNode (A1)││SpatialNode (B1)││SpatialNode (B2)│
└───────────┬────┘└───────┬────────┘└┬───────────────┘└┬───────────────┘
           ┌│─────────────│──────────│─────────────────┘                
           ││            ┌│──────────┘                                  
┌──────────▽▽┐┌──────────▽▽┐                                            
│Listener (2)││Listener (1)│                                            
└┬───────────┘└───┬────────┘                                            
┌▽──────────────┐┌▽──────────────┐                                      
│SpatialNode (2)││SpatialNode (1)│                                      
└┬──────────────┘└┬──────────────┘                                      
┌▽────────────────▽┐                                                    
│Output            │                                                    
└──────────────────┘                                                    

(Sorry the grapher made it all twisty.)

That is, each sampler should be routed to an independent spatial node for each virtual listener you have. You can then re-emit these listeners however you like, but in the above graph each listener produces one output, which is itself spatially processed with respect to the player.

If you had a bit of familiarity with bevy_seedling and Firewheel, this would be quite easy to set up. Although it might be tricky when you're starting out.

[hard plaza]

oh wow, thanks for the in-depth info!

[ionic sedge]

So indeed it's certainly possible! And maybe we could generalize some of bevy_seedling's pool behavior to make this sort of thing easier

[ionic sedge]

Okay @static quest I've put together a small list of recommendations for Component and Reflect in Firewheel. I'll make a little PR for that. Once that's in, I think everything's good for another publish from my end!

[short fossil]

While on the topic of derives, I noticed that quite many structs don't derive Debug, which was a bit annoying for me while debugging my code

[ionic sedge]

ya that's something that I think could be improved as well

[short fossil]

Stuff like ProcBuffer, ProcInfo, etc

[short fossil]

meanwhile me with 50 listeners:

[static quest]

noted!

[obsidian tusk]

How are you handling that, out of interest? Custom nodes?

[short fossil]

Yep. I have a node that sends the input it gets back to the ECS so the AI can process it in a system

[static quest]

What types specifically did you feel needed Debug derives?

[ionic sedge]

I've actually wanted Debug implementations for the events for a while now myself. You can't just derive it because of the trait objects, but manual implementations aren't actually bad at all. I'd recommend expanding a Debug macro on an enum to get an idea of what you need.

And in the manual implementation you'd just provide no information about the type erased variants I think.

[static quest]

If you want to send a PR for that, that would be appreciated!

Going through, I noticed I forgot to add Debug and Clone derives for some of Symphonium's types too. I'll fix that.

[short fossil]

Everything the processor and nodes get as params, basically

AKA the node and the node constructor 😉

[static quest]

Alright, I added Debug and some other derives to the nodes. Now just waiting on the PR to add Debug for events.

[rapid garnet]

maybe it would make sense to add the wayland feature to the list too (in the readme)

[static quest]

Wayland feature?

[ionic sedge]

for seedling

[static quest]

Oh ok

[rapid garnet]

yea sorry ment for the seedling readme^^

[static quest]

Whoops, should have tested before pushing https://github.com/BillyDM/Firewheel/actions/runs/18854037381/job/53797322662

[ionic sedge]

Also I realized I could put that list in the readme and rustdoc in a details section, which should help make it way less intimidating.

[static quest]

Ok, it was an easy fix.

Oh yeah, and what should we do about the bevy dependencies being on an rc version? Is it okay to use the full version now? https://github.com/BillyDM/Firewheel/blob/06502a95c0c42c502aaa3653af41077ac347caff/Cargo.toml#L173

[ionic sedge]

ya

we can bump them since it's published

at 0.17

[static quest]

Also for bevy_platform, we could probably get away with just setting it to version = "0". The stuff we use is not likely to change.

Or do you think that is a bad idea?

(If it is then I'll also need to change that in rtgc)

[ionic sedge]

I'm not sure. I don't know what people typically do here. If you don't mind the maintenance burden, I'd probably stick to a specific minor version. At least for a little while here, Bevy will want Firewheel to update for every Bevy version anyway 😅

[short fossil]

that's cheeky 👀

I also wish there was a stable bevy_reflect

[static quest]

I guess the annoying thing is that I would have to keep bevy_platform in sync for both Firewheel and rtgc.

Alright, bevy dependencies have been bumped!

[ionic sedge]

https://github.com/BillyDM/Firewheel/pull/88

[static quest]

Oh, I just realized that NodeEventType::downcast might actually deallocate. https://github.com/BillyDM/Firewheel/blob/ee9a5270fa352ee9e4287dfa72421788f77c3ff1/crates/firewheel-core/src/event.rs#L119 Would the correct way be to return Option<Box<T>> instead of Option<T>?

[ionic sedge]

Hm, it seems quite tricky to fully abstract over.

We might just give up in this case and return to an OwnedGc<Box<dyn Any>> . It would degrate the downcast experience since to achieve the same thing, the user would need to store Option<T> themselves. But nothing really comes to mind as a workaround.

[static quest]

I think I figured out a solution (and this is what the convolution node is doing anyway). Instead of returning the raw value, we have a downcast_swap method that swaps the contents of the custom event with a user's value.

[ionic sedge]

ya that's probably a good solution

it's not hard to get the current behavior if you really need it either way

[static quest]

Ok, I just went ahead and changed it.

https://github.com/BillyDM/Firewheel/commit/aca031d0298b43282124e08ea5f50de19c136043

And with that, were there any more issues needing addressing before I publish a new version of Firewheel?

[ionic sedge]

i think that's all the immediate stuff that im aware of :)

[static quest]

Oh actually, playing around with the visual_node_graph example, changing the parameters on the freeverb node don't seem to do anything.

Oh, it was simply just missing the params.update_memo function in the UI.

Hmm, should I make it version 0.8.0, or should I go up to 0.9.0? There have definitely been a lot of breaking changes since 0.8.0-rc.1.

[ionic sedge]

Hm, yeah it might be nice for anyone stuck on 0.8 for whatever reason if you publish a full minor bump

[static quest]

Alright, Firewheel version 0.9.0 has been published! 🎉

[ionic sedge]

Awesome, thanks! I should be able to get a new seedling version out tomorrow morning

[short fossil]

Oh hey, that means we can finally publish bevy_steam_audio 😄

0.1 will not have any docs

[static quest]

Oh yeah, I never finished the equalizer node. That can always be added later of course.

And I remember someone saying they had an implementation for a compressor we could use.

[stoic igloo]

I’m waiting for clack to go public for CLAP Node. There, I have my compressor.

[obsidian tusk]

There’s already that limiter node with configurable A/D/headroom, that’s already a compressor 🙃 The only things you’d need to add are a configurable ratio + maybe sidechaining so you can do ducking (e.g to quiet the rest of the world if a character is talking)

A limiter and compressor should probably share an implementation, with the limiter having a more limited (ha ha) interface and different defaults

[stoic igloo]

I’m reminded my compressor is limited. It’s ok.

[ionic sedge]

okay v0.6.0 is up

@short fossil bevy_steam_audio should be unblocked :)

[static quest]

@ionic sedge I should probably add a step in the CI to check no_std compatibility. I'll go ahead and do that real quick.

[ionic sedge]

I think I have a no-default-features run that just builds my crates. It's pretty easy to accidentally make a crate fail to compile for a number of reasons related to the default features (not just no_std), so it's been a decent catch-all for me.

[short fossil]

neat!

I see you also included the negative duration fix. Thanks!

oh, where did the downcast method go?

[ionic sedge]

ya we had a discussion about this earlier

wait

[short fossil]

if let Some(source) = event.downcast_mut::<audionimbus::Source>() {
    self.source = Some(source);
}

[short fossil]

this is the context in which we used that

[ionic sedge]

whew i got there

Yeah so basically we can't easily abstract over it without deallocating the box holding the trait object.

If you want this, you'll have to either:

1. Move to a swap API (doesn't really work here)
2. Just wrap the types in an Option and take (sorry!)

We may be able to provide this with a fancier set of traits, but the simple approach we took wasn't really appropriate.

[short fossil]

alright, take it is then

now how did I manage to trigger this while upgrading

oh, ogg feature in seedling

[ionic sedge]

it might have previously worked due to bevy_seedling not disabling symphonia's default features

[short fossil]

yep sounds like it

alright, publishing a 0.1.0 of bevy_steam_audio now

no docs because I couldn't find time

hehe

@ionic sedge are you still planning on adding that reflection mixer node? No pressure

[static quest]

Actually, it looks like he can use the new swap API here?

[ionic sedge]

ya definitely, i've just been quite limited lately myself!

[ionic sedge]

does it work if the target is an option that has no guarantee of containing the actual value?

[short fossil]

ah yep

event.downcast_swap::<Option<audionimbus::Source>>(&mut self.source);

this?

[static quest]

Correct

[short fossil]

Ah, or even event.downcast_swap(&mut self.source);

[static quest]

Yeah, that should work too.

[short fossil]

alright 🙂 again, no pressure at all!

[static quest]

Huh, weird. https://github.com/BillyDM/Firewheel/actions/runs/18881873317/job/53886928385

[ionic sedge]

ya ig they're mostly not required

[static quest]

But it fails to compile in no_std if I remove them.

Oh wait a minute, maybe not? It used to fail to compile without those imports, but now it seems to be working?

Did something change in a recent compiler version?