#Better Audio

11077 messages Β· Page 12 of 12 (latest)

slate scarab
#

Yes, it looks like it's all good!

slate scarab
#

Ah, well it passes my tests anyway. It seems like it's now terminating the players early though, before they've completed playback blobthink

#

I think I'll stick to 806496c for Rust Week collaboration and then I'd be happy to dig into it again later!

dusky mirage
#

Huh, that is odd. The logic should be exactly the same, but I'll double check it.

slate scarab
#

Sorry for not gathering more info β€” time is quite short this week!!

dusky mirage
#

Oh actually, I may have noticed something. Let me try a potential fix.

slate scarab
#

Oh let me be clear β€” I believe my pool code is observing the apparent finished ID well before it’s actually complete.

#

I can double check exactly what’s going on later.

dusky mirage
#

Does it happen when pausing/resuming, or does it just happen when playing a sample?

dusky mirage
#

Ok, so I think I found an issue with my new approach. The logic completely breaks down when pausing/resuming and using scheduled events.

I might have to mull over the best way to approach this.

#

Ok, I think I might have an idea of how to fix the reliability problem! Give me a minute.

dusky mirage
#

(I kind of want to get this fixed before Rust Week so we don't have people complaining about sample playing being unreliable.)

dusky mirage
#

@slate scarab Alright, so I've been doing some thinking and I think that trying to use the "ID" of the play parameter to detect when a sequence is finished is unreliable for a couple of reasons:

  1. Notify events aren't necessarily gauranteed to have increasing IDs because scheduled events could be sent out of order.
  2. The node pool state could get out of sync with the user's state since it has its own copy of the first node state.

Taking a step back, I'm worried that the node pool API might just be trying to be too generic. Having a "node pool" doesn't make much sense outside of a game engine context, so it might make more sense to make the node pool API specifically tailored towards games.

But then that got me thinking of exactly what it is that Firewheel is trying to achieve. Breaking it down, I think there are actually two entirely separate layers of concern at play here:

  1. The low-level audio graph engine
  2. A data-driven API tailored towards games

As an analogy, the low-level audio graph could be thought of as the low-level imperative/retained-mode layer of a GUI engine, and the data-driven game API is like a declarative/immediate-mode layer on top of that retained mode engine. (And that is kind of what bevy-seedling is doing.)

And there exists a fundamental conflict with the way the audio graph wants things to work and the way that a game engine wants to work. The audio graph wants to treat everything as a continuous signal flow between nodes, while a game engine wants to treat each individual sound played as its own distinct entity. I still think it could be done, but I think it could be worth taking a step back and figuring out the best approach to tie the two together.

#

I think it would be worth creating a short design document detailing the complete user-facing API that the game engine wants and the exact behavior it expects from the underlying audio engine, including details like scheduled events, how to create automation sequences or other sequences (i.e. playing a sequence containing a bunch of short sounds in quick succession, like a machine gun sound effect), cancelling sequences, pausing/resuming, musical sequences/rhythm games, and chains of effects.

Like one idea I had is to attach a sort of "sound entity ID" to each event. Events with the same ID are considered to all be part of the same "sound entity". And then the audio graph engine itself would be able to better detect exactly when a particular sound entity starts and ends.

dusky mirage
#

Another crazy idea I had (that would be a lot of work, but it could be worth it in the end) is to rework the engine to use a very different completely data-driven approach. Essentially, instead of having a low-level imperative API where the user has to manually spawn nodes and connect them, the audio graph could automatically do it itself via a declaration of the overall desired audio graph structure. And instead of having node pools and whatnot, the audio graph would instead spawn/despawn nodes on the fly for each "sound entity" played. (I was against this approach at first, but now it kind of seems like the only way to properly design a game audio engine. I think it might actually be possible without having to allocate/deallocate too much by re-using old allocations sent back from the audio thread.)

#

Oh, I had another idea. Instead of node pools, we could instead have a special "pool" node that has events containing the node processors of the sound entity itself, along with a mini "sub graph" of buffers connecting them. That way, entire "sound entity sub graphs" could be scheduled to play back a sequence at a specific time.

dusky mirage
#

Though of course we should think it through first before trying any new crazy ideas.

#

Another easier solution could be to add an extra event type to the sampler node which sends an ArcGc<AtomicBool> to the node on each new sequence as a sort of "handle" to that sequence. The sampler node then sets this flag to false when the sequence has finished.

slate scarab
#

@fiery surge look here!

#

Our current asset loader.

slate scarab
#

And here's a repo with a minimal seedling example. Feel free to work from here for review or PRs!

steep dove
#

Ok, so if the asset server produces something that implements symphonia::io::MediaSource, we can probe that (symphonia_core::formats::probe::Probe::probe) to get a Box<dyn FormatReader> which borrows the underlying MediaSource, and exposes seek() and next_packet() methods, that can produce chunks of audio for decoding.

#

@slate scarab

#

the trouble will be that we need to keep the MediaSource alive for the lifetime of the FormatReader.

#

and that may require a small amount of unsafe code to exploit how asset handles work, maybe

slate scarab
steep dove
#

Packet (returned by next_packet()) is Send + Sync. So it seems like the audio thread should hold the FormatReader, and offload decoding packets to other threads (with some sort of ring-buffer for receiving decoded data back from workers in a lockfree way)

#

@river marsh someone who is good at audio please budget my family is dieing

slate scarab
# 
// 1-to-1 graph
let volume = commands.spawn(VolumeNode::default()).id();
let high_pass = commands.spawn(HighPass::default()).id();
let volume2 = commands.spawn(VolumeNode::default()).id();
let low_pass = commands.spawn(LowPass::default()).id();

commands
    .entity(low_pass)
    .connect(high_pass)
    .connect(volume2);

commands
    .entity(high_pass)
    .connect(volume);

commands
    .entity(volume2)
    .connect(volume);
# 
commands.spawn((
    SamplerPool(MusicPool),
    sample_effects![SpatialNode::default()],
));

commands.spawn((
    MusicPool,
    SamplePlayer::new(server.load("my_sample.wav")),
    sample_effects![SpatialNode::default()],
));
slate scarab
#

Okay taking a look at on-the-fly decoding!

slate scarab
#

Okay I took a look and the simple implementation is not easy!! Yay!!

dusky mirage
dusky mirage
slate scarab
dusky mirage
slate scarab
#

I'm not convinced the ID check needs to be > anyway. != should sufficient, and it's what I've been using.

#

The game thread can keep track of which play event it's looking for, and once the finished ID corresponds to the play event, that is sufficient for completion detection.

#

The way it's managed in seedling should prevent desynchronization.

#

Note that we've also talked about this more dynamic approach recently, both here and in person! It's not necessarily functionality that Firewheel needs provide itself, except for making recompilations highly efficient.

#

We can (almost trivially) manage dynamic groups of sounds in the ECS. We can also cache arrangements of effects and connections to minimize recompilations in the common case (many sounds played with a similar set of effects and connections).

#

Combined with a configurable voice limit, this should on average have the same performance as rigid pools while being much more flexible.

dusky mirage
#

Though I guess I don't know the exact logic of your pool implementation. What exactly is the bug we're trying to fix again?

dusky mirage
dusky mirage
#

@slate scarab So are you not using any code in firewheel-pool at all?

#

Ok, by the looks of it, you aren't.

I'm not really happy with the API of firewheel-pool anyway. I think instead of trying to make a generic solution, it would be best for any game engines to just implement their own custom-tailored node pool system.

dusky mirage
#

@slate scarab Alright, let me know what you think of it now! (I also got rid of the worker_score method since it's likely that users will probably want their own special logic with custom priorities and whatnot.) https://github.com/BillyDM/Firewheel/pull/136/

dusky mirage
#

Symphonium has been updated to use symphonia 0.6! πŸŽ‰

dusky mirage
#

Whoops, made a mistake with u24 format on big endian systems, let me fix that real quick.

#

No, Codeberg is down again 😭

#

Well, I was still able to publish the fix to crates.io.

#

Ok, Codeberg is working again!

drowsy dome
drowsy dome
modern orchid
#

Hey, I wasn't sure where to ask this exactly, but seeing as rodio is planned to be removed, I was wondering if updating to its new versions would still be accepted.

I'v made a fix in its spatial audio support and now am looking to bring it to bevy (or decide not to), after rodio would get next release.

pine pelican
modern orchid
#

Ah thanks that is probably better place for it πŸ™‚ Ok then I will probably try to contribute version update when it is available.

dusky mirage
dusky mirage
#

@slate scarab Just checking up on you! 😁

slate scarab
#

Oh yes hiii! Are you still wondering about the pool code? I'd be happy to take a quick look in the morning! But yes, I can confirm that seedling isn't using it.

dusky mirage
# slate scarab Oh yes hiii! Are you still wondering about the pool code? I'd be happy to take a...

Yeah, I actually decided to just remove the firewheel-pool crate in the PR altogether since I'm not really that happy with the API anyway (it's probably better for game engines to just implement their own pool system instead of trying to make a generic one-size-fits-all solution).

I mainly want to make sure that the other changes in the PR work. Namely the changes to how the "worker score" works and also if the new SamplerState::playback_finished method works like how you want it to (or if you still need the last_finished_playback_id. https://github.com/BillyDM/Firewheel/pull/136

#

(For the "worker score", you now just read the data from SharedState::current_processor_state(). The two most notable fields are playback_state and playback_age_frames.

My reasoning is that I figured it would make sense to let the game engine decide how the "worker score" should work. Like it could also take into account the "priority" of a sample or its distance, volume, etc.

slate scarab
#

Yeah bevy_seedling right now simply uses the worker score in addition to its own scoring, such as from explicit priority or other factors.

#

I've had much less time than I thought I would recently, so I still need to do another round of checks on the changes. Sorry for the delay!

dusky mirage
#

It's fine, no rush!

rapid musk
#

Hello! I'm curious about how the upstreaming is going and referencing this comment #1236113088793677888 message

Since 0.19 is in the release candidate stage, does that mean that current ongoing work is clean up and preparing for upstreaming in 0.20? I don't see an issue for upstreaming Seedling and Firewheel on the Bevy repo, is it too early for that?

slate scarab
#

We’re more or less still blocked on maintainer attention. The two crates really need approval and/or guidance from cart.