Better Audio

[slate scarab]

oh yeah don't worry I'm keen to make sure it's all good

[rapid hedge]

Yes, it is immediately obvious that it's way better!

The bevy_audio version is live at https://janhohenheim.itch.io/foxtrot

And you'll notice that it clearly suffers from stuttering

In contrast, the seedling version has no stuttering at all.
Well, until I leave it on for a while in the background and the fixed timestep goes like "ooh let me run 20 physics updates this tick to catch up"

Then it stutters

[dusky mirage]

I'm working on it right now! 😁

[rapid hedge]

But that's not your fault but mine haha

The entire game lags in that moment

Should pause the game when focus is lost

[rapid hedge]

Oh thanks!

Do you need some info on what's wrong?

It's hard to describe, but the NPC is clearly moving to the right of me but the audio seemingly throws a dice on which channel to use 😅

[slate scarab]

oh yeah to be clear, it might have something to do with.... coordinates or something, idk, i'll be investigating in a minute

might just be my fault

[dusky mirage]

Well, my previous math was wrong anyway.

[rapid hedge]

I can't stress how huge this is btw.
All my jam submissions suffered from about 1 minute of stuttering audio on weak machines 😅

And now it seems like that issue is just entirely gone

[slate scarab]

Oh really? I'd actually still expect it to not stutter at all. It should be totally unaffected by the main thread (unless it slows your entire system down so much that it affects other threads).

[rapid hedge]

Seriously, great work y'all. This is amazing! ❤️

[rapid hedge]

Good point. I'll try again

[rapid hedge]

I played a bit around and it did one weird thing

I hovered over the buttons to check if the audio backend was working, and nothing played

so I hovered some more

and after like 2 seconds, all queued hover SFX played at once 👀

This was right after the Wasm app started

[slate scarab]

oh interesting
i suppose that could happen if the Wasm stalls for a moment, but it sounds like it wasn't?

or i mean, the main thread Wasm

[rapid hedge]

The UI was reacting correctly, so I don't think so

[slate scarab]

(the audio thread, again, should never stall or stutter)

[rapid hedge]

I can try if I can reproduce it later

currently leaving it open in the background to accumulate dt haha

[slate scarab]

if im lucky i might see it too here

[rapid hedge]

Can't reproduce the stutter from before

Chalk it up to Wasm being weird

Let's try the stall now

I can reproduce the stall!

OOOOH it's about focus!

When I start the game without clicking anywhere, nothing plays

Then when I click the first time, all queued audio plays at once

[slate scarab]

oh ya that's low key expected

[rapid hedge]

It's autoplay being blocked by the browser, right?

Then that's fine

On itch you'll have to press "play" anyways

[slate scarab]

basically yes -- although we could probably improve the detection? it seems like some clicks get eaten up

[rapid hedge]

Maybe worth adding to the FAQ somewhere

[slate scarab]

Yeah, in addition to some improvements to the mechanism.

[rapid hedge]

To be clear, I cannot reproduce the physics stutter either

Since I just discovered that I actually thought of pausing the game on focus loss

So idk what went wrong there haha

[slate scarab]

(the CLI actually also does this with the JS shim, and it might be slightly more robust)

[slate scarab]

Ah, yeah I know what the spatialization issue is. I'm not adjusting for the orientation of the listener.

Obviously that needs to be handled, but as a side note -- should we have an orientation lock for spatial listeners?

That could be useful for 2D games in particular, where you often want spatialized audio to respect the distance to the player, but not necessarily their orientation.

[crisp walrus]

I found it much better for my recent jam game but still stuttered on slower devices in the browser.

I'm really enjoying playing around with custom nodes though in firewheel though 😄😁😁

[slate scarab]

oh but that was because you weren't running with firewheel-web-audio though, right?

(that's the secret sauce)

Overall, bevy_seedling should be around 2-3 times faster than bevy_audio (thanks to Firewheel), but that's not fasterer enough to overcome the browser stutters in single-threaded mode.

(Because, to be clear, there's really no speed that would be fast enough. Sometimes frames just take too long and block audio.)

[crisp walrus]

I just imported bevy_seedling and went nuts, so ... I guess I probably wasn't using web audio? 😄

[slate scarab]

ya there's a few extra steps that are a little cumbersome at the moment

but I'm planning to make the web audio backend toggleable with a feature flag, so it should be much easier to use in the future

[crisp walrus]

yep. Overall though bevy_seedling was lovely to use 🙂

[slate scarab]

@rapid hedge if you have a moment, please feel free to checkout this PR: https://github.com/CorvusPrudens/bevy_seedling/pull/41 . It should resolve the issues you're seeing. If it's all good, I can publish a patch version with these changes since they're pretty important.

[rapid hedge]

Oh, I didn't see the ping until now for some reason

thanks! Will try now!

[rapid hedge]

Hurray, that fixes all spatial panning issues on my end

[rapid hedge]

Next step: release it on itch.io and see how it's working

[rapid hedge]

Here's my overall experience of having migrated to bevy_seedling from the point of view of a user who's pretty inexperienced in audio:

• I don't currently use any cool features, but still there are two major selling points for why I wanted to migrate to bevy_seedling:
  • stutterless audio in web (achieved )
  • easy settings (e.g. bevy_audio's GlobalVolume doesn't respect already running tracks, turning my settings code into )
• The node system is genius. I don't know if this is standard stuff for audio engines or something brand new, but it feels really really powerful. Just looking at the API opens up ideas in my mind about what I could do.
• Auto-despawning the sample players like in bevy_seedling is the correct API and IMO it's a footgun that bevy_audio doesn't do that.
• I can easily see how the new features would be useful for me in the long term. For example, how would I implement hearing a muffled voice from another room in bevy_audio? No clue. In bevy_seedling, I already have some ideas by just looking at the API 🙂
• There are some tiiiiny rough API edges that Corvus opened issues for after I ran into them, but that's all peanuts. Nothing blocking at all imo.
• The API feels very Bevy-y. The "tone of voice" of the Bevy API in general (not just audio) was kept.
• The docs are pretty good, especially compared to bevy_audio which often states things in very unhelpful terms
• That said, for me as an absolute noob, I feel like it sometimes assumed I knew the basics of audio, when in fact I didn't. Might be a skill issue on my part 😄
  • For example, I wasn't aware of basic knowledge like that 0 dB = full audio
  • Is it obvious to a beginner what a "sample" is? A "sampler", a "main bus", a "sampler pool"? "routing"? All of these feel like lower-intermediate terms, and not suitable for total beginners.

• The error messages are rather unhelpful until you already know how the crate "works". There's very little guidance on what I can do as a user to fix an error. Especially the one for sample_effects![SpatialBasicNode::default(), SpatialScale::default()] feels downright arcane relative to how fast someone would run into that mistake.
• The examples are nice! They weren't all-encompassing enough for me to just copy-paste all my functionality, but they gave me some great starting points. The two main ones I used were:
  • settings_menu.rs: Incredibly useful for learning the API of the crate in general.
    • To be fully representative of a real settings menu, the perceptual volume converter we discussed needs to be implemented and showcased there
  • spatial_audio.rs: Simple example to show the spatial API without any bells and whistles. Just what I needed!
  • It seems like spatial audio wasn't really tested in 3D contexts before, so maybe this could be a pointer for how to further improve / extend the examples 🙂

In my opinion, using bevy_seedling was a much more pleasant and productive experience than bevy_audio. From my own LIMITED user perspective, I would upstream bevy_seedling in a heartbeat. All of the improvements I mentioned are not blocking, mostly not better in bevy_audio anyways, and can easily be improved as part of the upstreaming process

@celest whale I know you like reading these ^

[celest whale]

That said, for me as an absolute noob, I feel like it sometimes assumed I knew the basics of audio, when in fact I didn't. Might be a skill issue on my part 😄
This would be really helpful to fix

cc <@&1064695155975803020>

Sounds lovely overall though

And "better than bevy_audio" is not a particularly high bar to clear

[rapid hedge]

Also want to mention how lovely the people in this channel were at helping me out 🙂

[celest whale]

It also seems like it will be a) more actively maintained and b) better architected in the long-term

[slate scarab]

I feel like it sometimes assumed I knew the basics of audio
I actually just added a glossary to the next release a couple days ago Sounds like I should add a few more entries!

[oak walrus]

wrt. naming, I would say "Sample" is on the same level as "Texture" or "Mesh" is in graphics -- it's a term fundamental enough that you're going to encounter it pretty quickly in your game dev journey. I don't think most people need to know exactly what it is and how it works, just how to use it. "Bus" and "routing" are audio mixing terms, slightly more specific and to it should not be expected that people already know what it is. The concept of pools is rather specific to seedling, but it ties to the concept of managing the amount of sounds playing at the same time, which almost all audio engines do

Maybe the docs team can chime in with experience on documenting the graphics engine how to best approach these prerequisites

[rapid hedge]

Alright, Foxtrot + bevy_seedling + firewheel web audio is now on itch! https://janhohenheim.itch.io/foxtrot

If you want to try it, make sure you've enabled WebGPU in your browser

As you can hear for yourself, the audio is buttery smooth, and the spatial audio works exactly as intended

(I forgot to lower the panning again after debugging though, so it's at 1.0 right now lol)

[slate scarab]

Foxtrot would be a lovely way to prove out some higher-level audio features btw! I might make a fork to test out all my stuff. One thing that comes to mind is reverb zones. I’d also love to try out my recent HRTF integration in 3D.

[rapid hedge]

HRTF would be amazing 😄

Feel free to upstream any cool features you try out into Foxtrot so they can continue being tested there

I'll just let you know if any migration breaks it 🙂

[vocal dock]

Though that assumes a free space spread of sound.

Inside a room where the soundwaves going up/down get "recycled" you'll probably get closer to 1/r, depending on the damping of the room.

[oak walrus]

Attenuation always assumes no obstacles, otherwise it's the domain of acoustics and you are definitely not going to get any specific falloff. Reverb volumes are what you add to fill up the space when it's not empty

[rapid hedge]

If y'all implement something like this, I can easily set up a little integration with TrenchBroom so that you can create little zones like this for testing 🙂

[oak walrus]

By reverb volumes I don't even mean anything ray traced or attempting to physically transport sound waves, mainly reverb volumes are automatic on/off switches for a reverb effect when the camera enters it

The reverb is a basic algorithmic reverb with artistic controls, not physical ones

[slate scarab]

ya this would be great

[rapid hedge]

Nice! Ping me once you have something to test and I'll integrate it so that you can just click around to create zones 🙂

[steep dove]

There is a partial rust port of steam audio in the form of phonon_rs

Steam audio has a bunch of complex raytraced techniques

Any of you audio experts planning to look at stuff like that after the new audio backend and api settles a bit?

[limpid mason]

Just a few days ago I read the Godot docs for audio and it explains a lot of things for beginners, including the decibel scale:
https://docs.godotengine.org/en/stable/tutorials/audio/audio_buses.html
We could probably draw inspiration from that

From reading that, their strategy seems to be to adjust naming and APIs more to expert users, but explain all the concepts in the docs, which seems sensible to me

[slate scarab]

Where should we put this sort of expository documentation? It seems like the Bevy book would be a good spot (after our work here would be upstreamed of course).

However, I’m not sure I like the co-location that imposes. Some crates, like winnow embed little tutorials right in the rustdoc.

[slate scarab]

I would love to get this sort of thing going, but it’s a big task!

Probably one of the most impactful next steps would be getting a high-level animation system going. It would be super nice for fades, musical sequences, etc.

And of course, such a system would also be very helpful in the rest of the engine.

[limpid mason]

Yea it's a tricky question tbh.
I think the Bevy book is definitely a good candidate, we will need an audio section anyway. But docs.rs should also be considered, as it's very discoverable. But the question on where in the docs to put it is a tricky one.
I'm not too much of a fan of having doc-only modules, it's kinda weird, probably the book is then a better place.
But I'm curious to hear which ideas the Bevy docs people have!

[limpid mason]

Would be amazing to have things like Avian colliders integrate with spacial audio so it's all a unified system
(I say knowing nothing about audio or physics :D)

[slate scarab]

ya that would be one of the more difficult parts for me personally

I don’t have much 3D dev experience

so if i could just plug in avian types, that would be awesome

Steam Audio has occluders and volumes and so on. I’m sure these would map very naturally to Bevy and or Avian components.

[limpid mason]

Also makes for a good UX when you have to learn each concept just once instead of multiple times :)

[limpid mason]

Some time ago I saw this ambient sound design video and it was super interesting for an audio noob like me.
It explains some systems on how to spawn the ambient sounds in the world to make them sound natural, e.g. spawning sounds randomly in a bounding box around the player or spawning a sound on the closest position on a line, etc.
Would be really cool to have these features integrated in Bevy :)

[slate scarab]

Ya things like “random playlists” are a common game engine tool, and that should be super easy to make. In fact, we might be able to pretty easily provide a whole set of tools like in the video. Some of their techniques are both widely applicable and easily expressible in the ECS.

[vocal dock]

Maybe the bevy book could have a second book attached to it, which covers general Game dev adjacent topics?

So it would have a chapter on the basics of audio engineering, and I believe a chapter on light and colour and colour spaces would be also useful

It wouldn't be necessarily a "Bevy" book, but it would be a "Technical Topics for Game Devs" by Bevy et al. book

and it would be attached to the standard bevy book for discoverability, and perhaps docs could link there?

[slate scarab]

ya however we organize it, I think we should frequently link to pedagogical material when we use jargon

with the assumption that people may land on any arbitrary portion of the crate docs

[vocal dock]

Yeah definitely, if anything I tend to have the problem of not landing on module docs enough

I always end up at whatever struct or method docs of the API I'm trying to use are

[slate scarab]

okay I got a little release out with those fixes (v0.4.4)

[vocal dock]

so links from there are needed

[rapid hedge]

Plug: https://github.com/janhohenheim/bevy_shuffle_bag

I believe I should maybe upstream that anyways

It has the Tim Cain trick of ensuring the shuffle bag is not picking anything twice in a row even when refilling

[rapid hedge]

thanks a bunch!

[steep dove]

When you shuffle a bag, do you ensure that the first item is different from the previous last item?

[rapid hedge]

Bingo

[steep dove]

Neat

[oak walrus]

Spatial audio in its current form doesn't do any geometry-based processing like ray traced audio. This is a question to ask the Phonon (Steam Audio port) team

[slate scarab]

ya that's what they were talking about i believe

[slate scarab]

By the way @rapid hedge do you have any opinions about https://github.com/CorvusPrudens/bevy_seedling/issues/29 ? I wrote it a while ago, but now that I think of it, it's kinda convenient to be able to set a default spatial scale for an entire pool (without using the DefaultSpatialScale resource).

Conversely, it may also be convenient to not have to dig into the sample_effects! to insert the scale component like the issue suggests.

[vocal dock]

While I conceptually agree, I would warn that games run on very rough approximations of reality.
And different systems might want to use vastly different approximations of the same thing.

Ex: Rendering uses rendering meshes to describe the collisions of photons and your objects

Meanwhile a physics library uses colliders with totally different shapes for physics collisions

An audio library will very likely want to use its own colliders that are yet again different. (Most likely they're gonna be even more simplified than the physics ones because you can't really "hear" the shape being wrong.)

[faint wigeon]

okay got most of the way to getting avb for lasers working last night but needed make some changes to seedling and fire wheel (mainly asio support). hoping to get it working tonight ✨

[oak walrus]

cpal already supports ASIO though, and it's the backend for Firewheel, maybe they just don't expose the feature flag from cpal?

[faint wigeon]

yup! basically. it wasn’t too bad

you need a different cpal host which isn’t currently configurable

[slate scarab]

oh I see -- hopefully my in-progress changes don't disrupt your additions too much

[faint wigeon]

i’m so thrilled with this work in general and will rebase all day to keep up

[slate scarab]

i miiiight be able to incorporate everything I've been working on into a release this weekend, it's been near-release-ready for a while

(but if you run into anything in the meantime, feel free to make issues or PRs!)

[dusky mirage]

@slate scarab Alright, here is the new and improved spatial basic node! https://github.com/BillyDM/Firewheel/pull/61

Right now the default values are just a rough guess of what I thought sounded good. I would love some feedback on better default values to use. Especially from someone using it in the context of an actual 3D game.

[slate scarab]

oh nice, I just started updating to the latest main

I'm definitely liking the new signature

very consistent

[dusky mirage]

I have even put the models into an interactive graph:

• distance gain model: https://www.desmos.com/calculator/g1pbsc5m9y
• distance muffle model: https://www.desmos.com/calculator/jxp8t9ero4

[slate scarab]

also destructuring the audio buffers in the signature can fit in one line
which is kinda nice

[dusky mirage]

Oh wow, didn't know you could do that!

[slate scarab]

ya i don't do it often because it can make signatures hard to read, but i like it in this case

[dusky mirage]

Oh, does adding cpal = { version = "0.16.0", features = ["asio-sys"] } to your project's Cargo.toml not add that feature?

@slate scarab I also realized it probably makes sense to have the "amp epsilon" be a parameter in the spatial basic node, so I went ahead and did that.

Oh actually, I can make the "smooth_secs" a parameter in all of the nodes as well. The reason I had it in a separate config before was because originally smoothed parameters allocated a temporary buffer, but now that's no longer the case.

[slate scarab]

ya i'd say that's overall a good idea
the more things are in the parameters, the easier it is to manage abstractions like the sampler pools

there might be an argument for moving less important fields like this outside of the parameters, even if they technically could be a param, but i don't think it would really make them any less clear in practice

[dusky mirage]

Well making them a parameter would make them more flexible.

[slate scarab]

By the way, do you think it would be crazy to split up the spatial node into a panner and a distance attenuator? (Or I suppose at least create an additional standalone attenuator.)

The distancce attenuation would be very useful as a stage after the HRTF node I'm working on. Fyrox's implementation has a simple distance model, but I'd prefer to use this new and more flexible approach.

If we fully committed to a split, that would mean the typical spatial approach would have at least two nodes in series. Is that too far? At least in bevy_seedling, people wouldn't necessarily need to know this upfront -- we're providing a convenient default graph that would handle this.

Of course, we could also write the attenuation part as a standalone little struct and incorporate it directly into each spatial processor. No need for node-level composition in that case.

[dusky mirage]

Hmm, actually that does make sense.

And now that I think about it, we don't even need to calculate the gain_left, gain_right, and muffle_cutoff_hz values in the audio thread. That could be done using a helper struct in the main thread.

[limpid mason]

Fair point! But maybe they still could use the same component with the same API? So that you just add multiple child entities, some with physics colliders and some for audio colliders?
But I don't know enough about how spacial audio would work to determine if that actually makes sense :D

[dusky mirage]

(Although that might make animating those parameters more difficult, I don't know).

[slate scarab]

ya I'd definitely love to keep the current setup here

[vocal dock]

yeah you could probably reuse a lot of stuiff

[dusky mirage]

Ok, cool.

[slate scarab]

reflection, which I think we're going to increasingly rely on in Bevy going forward, is a lot easier if the parameter types have public fields without any invariants

[slate scarab]

but however we want to do this, it would be nice to be able to use the distance attenuator on its own in third-party crates

[dusky mirage]

Should the distance attenuator also include the lowpass muffling stuff?

[slate scarab]

ya i think so

that would still apply even with the HRTF

[rapid hedge]

Hmm good question. Pro:

• less chance to walk into the parenthesis footgun
• set a scale per pool
  Con:
• looks confusing imo
• makes me feel like I need to memorize what goes in the macro and what doesn’t

I'm tending towards leaving it as-is 👀

[slate scarab]

makes me feel like I need to memorize what goes in the macro and what doesn’t
ya that's true

[dusky mirage]

Hmm, on second thought, it might be a good idea to minimize the number of nodes in the graph, especially ones that are part of a "pool". Maybe it would be better to just share a dsp struct between the "spatial basic" node and the "hrtf" node?

[slate scarab]

ya that works for me too

[faint wigeon]

@dusky mirage can i be incredibly annoying and ask how architecturally difficult or what limitations would there be to support multiple input/output devices

(i realize this is moving far beyond the scope of what games need)

[dusky mirage]

Yeah, currently the backend is limited by CPAL. Though in theory it should be possible to create a "CPAL input node" similar to the StreamWriter node.

@slate scarab Alright, I separated out the distance attenuator dsp.

[slate scarab]

the firewheel-web-audio backend actually has nice, low-latency input capabilities

but it's limited to a single input device as managed by the browser, so it's not crazy or anything

[slate scarab]

In other words, you could probably fairly easily (?) write a backend that supports multiple ins/outs for specific use cases. Have you seen the backend trait or any (relatively) simple implementations of it?

[dusky mirage]

Are you happy with the new API of the spatial basic node? If so, I'll go ahead and merge it.

[slate scarab]

i was just about to take anothter quick look!

[slate scarab]

It may actually be a little tricky to plumb up inputs and outputs to the graph without weird latency consequences, but that may not matter for what you want to do with it.

[slate scarab]

ya so I think this is great -- the fine control will be very nice for those who need it
there's a very good chance we'll want to tweak things based on how everything feels in actual games

but I don't think we can sort everything out until we use it in practice

if you wanted, you could gather up all the attenuation fields into its own parameters struct. Obviously, diffing and patching derives work fine for nested structs, and reflection would still work great in that case.

probably the primary downside with this is that the field paths in the event representation would be two elements long, thus requiring allocations

(i.e. these fields https://github.com/BillyDM/Firewheel/blob/64d65cfb6b77603473ef7e15ee3b6087f2a7f0b4/crates/firewheel-nodes/src/spatial_basic.rs#L68-L174)

[slate scarab]

the advantage with using this nested struct is that we could drop it inside other spatial nodes very easily

e.g.

pub struct SpatialBasicNode {
    pub volume: Volume,
    pub offset: Vec3,
    pub panning_threshold: f32,
    pub downmix: bool,
    pub attenuation: DistanceAttenuation,
}

pub struct HrtfNode {
    pub offset: Vec3,
    pub attenuation: DistanceAttenuation,
}

I think in practice this extra allocation will be completely negligible.

(The vast majority of events for each will be offset patches.)

[dusky mirage]

Actually yeah, that makes sense. I imagine that the attenuation parameters won't get changed that often anyway.

[slate scarab]

And then the compute_values method can take a reference to DistanceAttenuation, which would reduce the verbosity there!

or whatever we want to call the type

[dusky mirage]

Yeah, I think DistanceAttenuation is fine.

Oh, and I imagine the "muffle_cutoff_hz" parameter could change quite often, so I'll keep that separate as well.

Ok, done!

[slate scarab]

ya I think that looks good!

[dusky mirage]

Ok, merged!

Are there any more outstanding issues with the API you can think of? If not, I might go ahead and release 0.7.0-beta.0.

[slate scarab]

hm, I thought I might have had something, but nothing comes to mind at the moment

[slate scarab]

Something I'm puzzling over is how to best encapsulate the generic backend for bevy_seedling, and more importantly how to make it easy to use the web audio backend.

The plugin looks like this:

pub struct SeedlingPlugin<B: AudioBackend> {
    pub stream_config: B::Config,
    // other, non-generic fields...
}

For the cpal backend, we have a cpal-specific configuration. For the web audio backend, we have a significantly simpler, web-specific configuration.

The plugin inserts the stream_config field into a resource. Any mutations to the configuration resource after startup will cause the stream to stop and restart with the new configuration. This is very convenient. It also means the plugin's generic parameter can't be fully abstracted or type-erased.

My question is -- should we make this an obvious choice by default? I could imagine a few approaches.

// Make the choice clear -- slightly annoying to switch backends
#[cfg(not(target_arch = "wasm32"))]
app.add_plugins(SeedlingPlugin::default());
#[cfg(target_arch = "wasm32")]
app.add_plugins(SeedlingPlugin::web_audio());

// A variation of the above
#[cfg(not(target_arch = "wasm32"))]
app.add_plugins(SeedlingPlugin::<CpalBackend>::default());
#[cfg(target_arch = "wasm32")]
app.add_plugins(SeedlingPlugin::<WebAudioBackend>::default());

// Obscure the choice, automatically selecting the web audio backend
// when a `web_audio` feature flag is enabled
app.add_plugins(SeedlingPlugin::default());

The last approach would change the type, but I think in most cases that would be fine.

[dusky mirage]

Would it be possible to put your web audio backend in the Firewheel repo? I imagine it would be useful for more than just Bevy.

[slate scarab]

oh ya definitely, it’s already completely detached from bevy

https://github.com/CorvusPrudens/firewheel-web-audio for reference

[dusky mirage]

Cool, you can submit a PR for that if you wish.

[slate scarab]

i should be able to get that going tomorrow

[dusky mirage]

Web tech is crazy stuff

It probably would have been unthinkable a decade ago that you could create a fully featured, high performance game engine with custom GPU rendering for the web without having to install any plugins.

I've heard some audio plugin developers are taking interest in webassembly as a fast, JIT-compiled language that can run on any platform.

[oak walrus]

Main problem is going to be real-time safety, off-the-shelf runtimes won't be made with that in mind so there is going to be some duplication of work here

There's also the inherent overhead of running a stack machine, even if JITed into machine code, it's not a zero-cost abstraction

Also because it's sandboxed by default, you need to design the whole API, how to provide the audio and event data, how to do GUI, etc.

[slate scarab]

idk I don't think the stack machine itself has very much overhead

[pure dove]

yeah, java bytecode is a fairly abstract stack machine and java JITs can still turn it into whatever representation they like and compile from there

[dusky mirage]

Ah yeah, that sounds like quite a hurdle.

[dusky mirage]

I didn't know webassembly even used a stack machine. I thought it was just a subset of assembly that got JIT-ed into pure machine code and then invoked like a function.

Though I guess I don't know entirely what a "stack machine" is either.

[slate scarab]

well, the Wasm instructions model a state machine, but i don't know how faithful JIT compilers have to be to it

but the instructions are like

i32.const 1
i32.const 2
i32.add

where the first two push to an implicit stack, and the add pops both off, adding them and pushing the result back onto the stack

https://developer.mozilla.org/en-US/docs/WebAssembly/Guides/Understanding_the_text_format#stack_machines

[dusky mirage]

Oh ok, so in essense it's how the JIT compiler translates this "virtual state machine" into actual registers?

[slate scarab]

well idk, probably depends on the context
I'm sure a lot of the time it actually will work with the real stack when there's a lot of parameters

[dusky mirage]

Makes sense. It seems to have been designed to be very flexible in the way JIT compilers can choose to optimize it.

[rapid hedge]

We're discussing asset-preprocessing over in #1278871953721262090

Does audio want anything?

If so, come say hi 🙂

[slate scarab]

ya

[slate scarab]

btw on the topic of opus -- do we even have a pure rust decoder yet? I think symphonia's stalled on it at least

the maintainer mentioned they wanted to tackle it, but they seem to have had little time for symphonia over the past couple years

[dusky mirage]

Oh, I didn't know symphonia's implementation was incomplete.

Oh yeah, you're right. I don't know how I didn't notice until now.

[slate scarab]

it's a sad file with nothing in it XD

[dusky mirage]

I guess I just never thought to test loading an opus file.

[rapid hedge]

Oh wow that was not hyperbole

Same with wavpac

[dusky mirage]

Considering the reference implementation is open source, I imagine it is possible to port it to rust. It's just someone needs to put in the time of course.

[slate scarab]

ya, seems to be the same with SOFA (for HRTF data)

well

[rapid hedge]

Is it C? If so I heard there are transpilers to Rust

[slate scarab]

that one doesn't have an open source reference, but

[rapid hedge]

https://github.com/immunant/c2rust

[dusky mirage]

Ah yes, libopus is written in C.

https://github.com/xiph/opus

[rapid hedge]

Oh great! I've never used c2rust, but I've heard people had great success with it 🙂

Would be curious if any of you wanted to try it out

[dusky mirage]

Ah, looks like someone has already transpiled it. https://crates.io/crates/unsafe-libopus

[slate scarab]

id rather reimplement it tbh

not that i would though, it would probably take a long time 😅

[rapid hedge]

The point of transpilers is that they allow you to write a test suite

You can then rewrite your heart's content and make sure it’s still working

[slate scarab]

that would be nice

maybe we should do that with libmysofa

[oak walrus]

I mean I think the aim of c2rust is exactly that: have a starting point so you can refactor the unsafety out bit by bit

Also now that it's in rust, you can run MIRI on it and probably catch some unsoundness issues to forward back to the C library even

And then, yeah, write regression tests, then hack away at a safe refactoring

The other issue might be licensing

[slate scarab]

sounds like it would be a very mechanical process for most of it, which isn't so bad

[oak walrus]

And how derivative is transpilation

[rapid hedge]

@slate scarab what’s the issue with libsofa's license? Quick googling tells me it’s BSD-3, which should be fine?

[rapid hedge]

Or did I misunderstand you? 🙂

[slate scarab]

libmysofa is an open source project with a workable license that operates on a subset of the SOFA format
I mean it does everything we need, so no worries there, but it's not exactly a reference implementation

[rapid hedge]

Aaah gotcha

[slate scarab]

and the format itself is one of them annoying pay-$100-for-the-privilege-of-looking deals

[rapid hedge]

How does it make sense to have an implementation be open sources but the format not?

[slate scarab]

[rapid hedge]

Doesn’t the implementation kinda imply the format?

[slate scarab]

nah i think C and C++ compilers are the same deal

[rapid hedge]

I see

[slate scarab]

the final specification for each is a licensed document, and probably helps fund the specification groups or something, even if the compilers themselves are fully open source

[oak walrus]

The C++ ISO reference document is indeed paid, I don't know about C

But given nobody actually looks at the document, I don't think they're getting much money from it anyway

And there's cppreference for open reference documentation too

[slate scarab]

can any cargo experts help me figure out why transitive dependencies of glam with versions 0.29 and 0.* don't resolve to the same version?

the 0.* group is somehow on 0.30, which causes problems for the 0.29ers

[dusky mirage]

Hmm, does cargo clean help?

[slate scarab]

ironically that's what caused the problem 😅 or rather, my lockfile convinced cargo that 0.29 was okay for all of them

[dusky mirage]

Oh, ok

[slate scarab]

but in CI or after a cargo update, it's diverging

My expectation would be that cargo would prioritize the most specific version and prefer a single version.

But that doesn't seem to be happening

[limpid mason]

Not what you're asking, but would it help to commit the Cargo.lock? Then CI would be forced to use the same version :D

[dusky mirage]

Yeah, I think that's specifically what Cargo.lock is for.

[slate scarab]

sure but wouldn't this happen to anyone depending on my crate as well?

[dusky mirage]

Oh, I don't know.

[celest whale]

This is why bevy doesn't commit Cargo.lock files

[slate scarab]

Oh actually I think this explains it https://github.com/rust-lang/cargo/issues/10599#issuecomment-1109775541

so that's a bit annoying 😅

I don't really want to force Firewheel to track the version of glam that Bevy does

[clear wasp]

Semver considers every minor version for 0.* to be different

https://www.lurklurk.org/effective-rust/semver.html

[celest whale]

Yeah, this is a nuisance

We're going to be upgrading to latest glam for 0.17 anyways it looks like

Despite how displeased we are with the getrand situation

[slate scarab]

It definitely worked before though. In other words, it seemed like it effectively opted into accepting technically semver incompatible versions. After some seemingly insignificant change in my dependency tree, though, it's now resolving in such a way that I'm ending up with two different versions. So

@dusky mirage would you be okay with sticking to Bevy's version of glam for the time being? At the very least, it's already feature-gated in Firewheel and totally optional, so it shouldn't have significant negative consequences for non-Bevy projects.

[slate scarab]

Oh to be clear, this would mean 0.* -> 0.29. And then 0.30 when Bevy releases 0.17.

[clear wasp]

You can have different versions in your tree just fine, you just can't use the types interchangably. Depending on which crates pull in the different versions and how they use them it might work fine or not.

[slate scarab]

When I say "seemed to work," I mean "there was only 0.29 in my tree"

If I properly understand the link I pasted above, I think this is just something that can happen sometimes.

[clear wasp]

Ah, I think that means that some other crate did a semver violation actually. Crates are not allowed to do a semver bump on a dependency that's part of their api surface in a non-semver bump.

Which is why cargo update should in theory always be safe if everyone follows the rules, but in practice it's often not

[slate scarab]

well 0.* might be a semver violation all on its own for cargo's interpretation of semver 😅 I'm kinda surprised it's allowed

[clear wasp]

This comment made me try to install c2rust, but after failing for 20 minutes I think it might be faster to just do the rewrite 😄

[slate scarab]

alas

[dusky mirage]

Sure, that works

[slate scarab]

cool, I can make a tiny PR for that

[dusky mirage]

Perhaps you can rename the "glam" feature to "glam-30" (or whatever version bevy uses). That way we can add a different glam version if we needed to.

[slate scarab]

Okay I've done the bulk of the work for bevy_seedling's next release. The pre-release work is on master now. @faint wigeon whenever you have a moment, feel free to check out the changes! The initialization / configuration is better, and starting to be more ECS-integrated, although it's still not perfect.

Most of the big items are covered in the change log.

[dusky mirage]

@slate scarab Merged your PR!

[faint wigeon]

yay!! excited to check. and also, the reason the multi device thought came up is just that the relationship between hosts and devices is actually pretty similar to the swirl we’re going around in rendering about supporting multi gpu and transitioning from resources to components

same api shape we also have adapters (hosts) and devices

taking a step towards being more ecs driven is awesome (:

i just tend to find that everything becomes more of a virtuous cycle when you lean into the ecs

[slate scarab]

I have thought about how you could technically have any number of separate audio graphs in the ECS at once. I'm sure we could find a way to make it ergonomic to specify which stream a node corresponds to.

[slate scarab]

Something that might be interesting to look into is a dedicated thread for handling time-sensitive scheduling and audio file streaming.

I've been thinking about how we could use the new precise event scheduling to create loop regions (or any arbitrary musical sequencing, really) that are controlled from the game logic. However, the edge cases worry me. If a frame randomly take extra long (and often it can be very bad, like a second or more), we might miss a loop point. That could be really bad for general musical synchronization!

If we had a separate thread just for scheduling, though, we'd pretty much eliminate this possibility. And, looking ahead to audio file streaming, we could also use it to reliably perform file operations.

Obviously this doesn't really work for Wasm (unless we do multi-threading :3 which you should do anyway for good audio), but given the overall API we've built up, it seems like it might be a promising approach in general.

Of course, we could always build out the sampler more (or create special sampler nodes) to handle this, but a semi-high-priority scheduling thread is a lot more general.

[oak walrus]

I don't know what that would achieve, tbh. You can replicate multiple graphs as a single one with multiple strongly-connected components. Specifically with synchronization in the case of driving multiple inputs and outputs. Scheduling a single graph is going to be easier than multiple ones, even if the graphs are independent, because latency is going to be an issue nonetheless. Unless there's something I'm not seeing, I don't think it's a good idea.

[oak walrus]

I would also say to not use the event system for that. The I/O thread and audio engine need to both know the looping points for streaming, but other than that, I don't think it's going to be reliable if gone through the event system.

[slate scarab]

In my mind, separate graphs would be for separate tasks. It's somewhat niche, but there's nothing fundamentally blocking such an approach.

It could be very useful for installations, for example.

[oak walrus]

Even for installations I think a single graph is better because of synchronization issues in case users use multiple graphs, yet expect both to be driven at once

I also don't think it would have any performance-wise benefits

Adding complexity for nothing, IMHO

[slate scarab]

People using multiple graphs would understand the consequences. A primary use case would be running multiple completely separate streams on many devices, while still sharing the same ECS core.

[oak walrus]

You can do multi-device on a single graph, get the benefits of running a single schedule, without any of the downsides or independent schedules

[slate scarab]

[slate scarab]

Anyway, this is not me suggesting we do streaming via events. Rather, I'm fairly sure a scheduling thread could perform double duty as a streaming manager.

[slate scarab]

Hm, so I ran into an issue where RealtimeClone wasn't implemented for (). But to be honest, I can't see where it's implemented for anything except ArcGc and a couple of the clock enums

(context: I wanted a Notify<()>, because I want to be able to trigger an action, but it has no associated data)

for some reason Notify<bool> worked though

[dusky mirage]

Oh yeah, RealtimeClone is not implemented for any primitives because it clashes with other implementations (I can't wait for negative implementations). Though I did forget about Notify<()>, I can add that.

The reason other primitives work is because I manually implemented them in the macros inside of leaf.rs.

[slate scarab]

It does? Hm, that’s annoying

i would like one (1) specialization please mr rust

[oak walrus]

We have specialization at home https://lukaskalbertodt.github.io/2019/12/05/generalized-autoref-based-specialization.html

[slate scarab]

ya i think bevy uses that for a couple neat things

[dusky mirage]

@slate scarab Ok, I've implemented Diff and Patch for Notify<()>!

[slate scarab]

oh we should probably also implement debug for a lot of the public types like NodeEventType
i haven't manually implemented debug for an enum in a while, but it should be super easy to just provide blank implementations for the erased variants

I can submit a pr for that at some point

[slate scarab]

alright the scheduling api is coming together
I'd say this looks pretty nice

fn precise_scheduling(
    node: Single<(&VolumeNode, &mut AudioEvents), With<MainBus>>,
    time: Res<Time<Audio>>,
) {
    let (node, mut events) = node.into_inner();

    // fade down
    node.fade_to(Volume::SILENT, DurationSeconds(2.5), &mut events);

    // and then back up
    let now = time.context().instant();
    node.fade_at(
        Volume::UNITY_GAIN,
        now + DurationSeconds(2.5),
        now + DurationSeconds(5.0),
        &mut events,
    );
}

the fade methods are just special-cased scheduling for the volume node
arbitrarily scheduling single events would look like

    events.schedule_absolute(now, &node, |node| {
        node.volume = Volume::SILENT;
    });

instead of scheduling individual events in the fade animation at a constant rate, we can use the just noticeable difference to determine how many steps the fade should take

this allows us to effectively guarantee that no zippering will be heard, even for very fast fades

(another neat benefit is that very long fades don't generate a billion events)

[slate scarab]

@dusky mirage do you think it would be practical to make NodeEventType implement Clone? I think this might significantly improve the robustness of managing scheduled events in the ECS. Let me explain the problem.

In order to maximize the chance of scheduled events arriving exactly on time, I send them a couple frames ahead of time when possible. I went over [why this is necessary a while ago](#1378170094206718065 message), in case anyone needs a refresher.

This presents a problem, however: I can't simply "render" the events to a Vec<NodeEventType> if I want to apply them to both the audio processor and the ECS. NodeEventType doesn't implement Clone. This is, of course, very desirable; the ECS should be kept in sync with the audio context, even if we schedule events. However, since I'm sending them a little ahead of time, they'll simply be gone by the time we need to apply them in the ECS.

My immediate solution for this was to just "replay" changes by storing a type-erased function that generates the correct NodeEventType. That way, I could get the events ahead of time and exactly when the ECS needs them. This mostly works, but generating these events is not idempotent -- Notify being the prime culprit.

However, if we could just clone the events, then they could be cloned and sent ahead of time to the audio engine, then later applied in the ECS. No room for subtle errors, there.

Of course, Clone would be problematic for the Custom variant.

Let me know if you have any better ideas, or if the problem isn't clear.

I may be able to work around this, but I'm not super confident it'll work in all cases.

[dusky mirage]

Hmm, yeah Clone would be a problem for the Custom variant.

In theory we could add a try_clone method that returns None if the variant is Custom.

[slate scarab]

And the custom variant is set up this way to transfer types that don't make much sense to clone either way, right?

[dusky mirage]

Correct. It's for sending, say, a new pre-allocated buffer to the node processor.

[slate scarab]

hm

we might get away with only cloning the param variant

[dusky mirage]

Do you want me to go ahead and do that?

[slate scarab]

Hm, I might test it a bit locally to verify my idea will work out

I think it's probably okay to mention in bevy_seedling's docs that special events like that can't be scheduled in quite the same way -- I don't think any of the current nodes use Custom yet, right?

[dusky mirage]

Correct

[slate scarab]

okay it seems to be working fairly well
hopefully i can wrap all this up soon in another release

[slate scarab]

Hm @dusky mirage do you think it's possible that an event with a timestamp "in the past" arriving simultaneously with timestamps that aren't in the past could cause the former to be dropped?

I'll be able to give more info on this as I diagnose. The use case here is scheduling playback for a sample that hasn't loaded yet. By the time it loads a couple frames later, the start time is in the past. On its own, this works great! Super cool to see. But if I add a fade in at the same time (scheduled for a VolumeNode down the graph), the play event doesn't seem to arrive.

Anyway, I'll see if I can find where this might be occurring in the scheduler.

[dusky mirage]

Oh yeah, I currently have it set to non-scheduled events taking priority. https://github.com/BillyDM/Firewheel/blob/df9f2e4403cdd4cb90ce092bac676662d37c84dd/crates/firewheel-graph/src/processor/event_scheduler.rs#L619.

We could add a flag to let the user choose if they want scheduled events or non-scheduled events to take priority.

[slate scarab]

well actually right now I have them all being scheduled -- not that I'll keep it that way, it's just easy to make sure everything lines up

[slate scarab]

definitely worth thinking about -- I'd like to feel out how unscheduled events mesh with the scheduled ones a bit more

so I'll have more to report there soon i think

[slate scarab]

oh interesting, im not sure it even should have worked before -- however, tossing in the other events seems to correct the sorting

Without any other events, the sorted events buffer looks like:

id: 2, time: Some(Seconds(InstantSeconds(0.9204761904761904)))    
id: 2, time: Some(Seconds(InstantSeconds(0.9204761904761904)))    
id: 2, time: Some(Seconds(InstantSeconds(0.0))

These three events are for a sampler node. They must be

1. the sample itself
2. a "paused" playback state, which I initialized the params with
3. the specifically scheduled "play" playback state

If these were sorted as I'd expect, the "play" event should really happen first, meaning it would be "dropped," effectively. I'll have to figure out how to solve that in bevy_seedling. However, when I toss a ton of unrelated events in (the VolumeNode events I mentioned earlier), the event scheduled for InstantSeconds(0.0) actually does get sorted to happen first.

In this case, I think we don't technically need the first two events to be scheduled, meaning I might be able to wrangle bevy_seedling into doing what I want here.

Removing the scheduling from the first two and swapping the priority of scheduled vs unscheduled did fix this particular issue. I wonder if it's the right approach overall, though.

[dusky mirage]

Oh, keep in mind that we are using sort_unstable for scheduled events, so events that have the same timestamp are not guaranteed to be in the same order that they were sent.

[slate scarab]

Hm, is it possible that clumps aren't always sorted? The comments mention that adjacent events for the same ID can be used to accelerate certain operations. That could maybe explain why this happens when no other IDs are present in the event queue.

[dusky mirage]

I suppose we could add logic to the sampler node for samples that aren't loaded yet. That seems like a common enough use case to warrant official support.

[slate scarab]

Hm, what kinds of APIs are you thinking?

Ideally, for a real project, you'd ensure the assets are loaded before the point that you need them, or you just stream long files like music. But for convenience and general robustness, we handle the case where samples aren't yet ready in bevy_seedling.

[slate scarab]

This is to say -- I think the problem has reasonable solutions in user-space already, but I suppose there could be something that would make it easier to handle.

[dusky mirage]

We could add a "play even if there is no sample" flag to the config and have it listen to when the sample parameter gets an update.

[slate scarab]

oh i see, yeah that would be convenient

[dusky mirage]

Oh, there is one complication. We would need to know the length of the sample up front.

(Or wait, maybe not.)

I'll work on it today and see if any complications arise.

[slate scarab]

In bevy_seedling right now, we don't actually "acquire" a sampler until the asset is loaded, so the primary concern for me is just making sure a Play event scheduled in the past would work with a sample that's scheduled much later.

In other words, the PlaybackState::Play and sample asset will be sent to the audio thread at the same time, but the former may be scheduled way in the past depending on how long it actually took to load the sample.

[dusky mirage]

I see.

I'll look into it more after breakfast.

[slate scarab]

And, to be clear, if the sample event doesn't have a timestamp and has a lower priority (is applied before scheduled events), it would work out in my case. But like I mentioned, it's hard for me to determine if this is good in all cases across all nodes and events.

It might be totally fine

But it's definitely clear to me that scheduling everything by default is more robust and would likely maximize correctness, at the cost of lots of scheduling and this particular PlaybackState / sample asset ordering issue.

(Scheduling everything in bevy_seedling by default, that is.)

[slate scarab]

Part of the reason for this is that uses don't have ergonomic control over when normal, non-scheduled diffing would occur in comparison to scheduled events. But if normal, end-of-frame diffing events were scheduled using the same timestamp as scheduled events in that frame, then the ordering becomes very clear and reliable.

[dusky mirage]

(Also, it is I, ClippyDM) 🤘

And now that I got the priorities out of the way, I'll look into the scheduling problem.

[dusky mirage]

Hmm, it should be as simple as setting the "sample asset" and the "PlaybackState::Play" events to occur at the same time. Is that what you're doing?

[slate scarab]

Well there's sort of two things going on here. The first is that, seemingly, the sorting isn't totally correct in certain circumstances like I mentioned here. These three events were probably pushed in this order, but the sorting should reorder the 0.0 event to happen first. However, this is their post-sorting order.

The reason three events instead of two are pushed in the first place is really just a consequence of bevy_seedling's overall abstraction here. I don't think I'll have any trouble special-casing the playback scheduling to happen how the user would want. In other words, I don't think I'll be forced to rely on something like:

We could add a "play even if there is no sample" flag to the config and have it listen to when the sample parameter gets an update.

I actually already have special handling around sampler nodes, given the pools and surrounding abstractions in bevy_seedling.

And also the desire to maintain a familiar interface for bevy_audio users.

[dusky mirage]

Yeah, and this might end up being a bit more complicated to implement than I first thought.

[slate scarab]

ya I probably wouldn't worry about that then

If you'd like a repro for the sorting issue, just let me know! I seem to be able to get it to reliably occur, so I should be able to create a minimal repro outside of seedling.

[dusky mirage]

@slate scarab Could you add the following to the end of prepare_process_block? https://github.com/BillyDM/Firewheel/blob/58511435339cd681f9e6062d8389f10dda525e2a/crates/firewheel-graph/src/processor/event_scheduler.rs#L507

dbg!(&self.sorted_event_buffer_indices);

for (i, _) in self.sorted_event_buffer_indices.iter() {
    dbg!(&self.scheduled_event_arena[*i as usize]);
}

dbg!(self.num_elapsed_sorted_events);

It should be that all three events get scheduled in the same process block.

Oh wait a minute, I might have made a mistake here. I think it's supposed to be < https://github.com/BillyDM/Firewheel/blob/58511435339cd681f9e6062d8389f10dda525e2a/crates/firewheel-graph/src/processor/event_scheduler.rs#L170

Yeah, it's definitely supposed to be that. That would cause events to not be sorted. 😅

Ok, pushed a change. Let me know if that fixes it!

Might want to run your benchmarks for event scheduling again. This may have caused events to not even be sorted at all!

[slate scarab]

although it did seem to sort the events in my testing just now if i had more stuff flying around, so that might have been an edge case?

[dusky mirage]

Ah yeah nevermind, if you were scheduling a bunch of events at random times, that would still trigger the "events_need_sorting" flag. But still, it wouldn't get triggered if the only events that were there each had a time less than or equal to the last one.

At some point I also want to do some benchmarking to see if using sort_unstable is faster or if plain' ol binary searching on each new event and then inserting it in place is faster. In theory events will typically already be in the same order that they were pushed, so pushing should be more common than inserting.

[dusky mirage]

@slate scarab Did that fix the issue, or is it still present?

[slate scarab]

ah sorry -- i was wrapping up work! I should be able to test within the hour

[slate scarab]

okay ya can confirm the latest commit resolves this case

I have another small fix queued up (sending Stop doesn't currently set the finish atomic with that Stop's ID) and the clone stuff. @dusky mirage would you potentially be interested in enforcing clippy lints in Firewheel's CI? I generally quite like it, personally, although it might be a bit of work to bring all of Firewheel's crates in line.

[dusky mirage]

I'm fine with that.

[celest whale]

Do y'all want a shoutout in Future Work for 0.17?

[slate scarab]

oh that would be cool!

[celest whale]

Feeling confident enough that we can port soon?

[slate scarab]

I think we’ll definitely be at a point where we can consider it soon! I’m working on incorporating Firewheel’s latest scheduling improvements into bevy_seedling, after which I can publish a big release with a bunch of goodies. The release will largely be a refinement of the API, so I feel decently confident in the overall direction and the capabilities of Firewheel, personally.

[celest whale]

Excellent 🙂

I should have bandwidth next cycle to help with the demo too, if y'all need some help there

[rapid musk]

Very cool! I'm really looking forward to the demo and the Bevy port. I frequently check Discord for any new activity related to this work, it's very exciting! It's also very interesting to read your discussions regarding design and issues that pop up. Thank you Corvus and BillyDM for all the work you've put into this.

[dusky mirage]

Yeah!

[slate scarab]

Okay @dusky mirage I think post-#63, I'll be ready for a bevy_seedling release. I think the Debug and clippy changes I suggested earlier could be done without any breaking changes, so we won't be blocked on those even if you make a new Firewheel release. No pressure though -- even another beta release would totally work for me!

[dusky mirage]

Though sometimes clippy does complain about function arguments, so I'll see if it complains about any that would cause breaking changes.

Oh yeah, merged!

[slate scarab]

oh ya i generally disagree with that lint actually -- you could easily disable any lints that you don't agree with or would cause breaking changes unnecessarily

[dusky mirage]

Are you using Hash for ArcGc anywhere? Clippy is giving me a hard error there.

I can just remove the hash derive if you aren't

[slate scarab]

I don't believe so -- let me check.

okay yeah looks like I don't rely on it anywhere

[dusky mirage]

Ok, looking at the clippy lints, there are a few places where we return an error of type (). I'll fix those.

[dusky mirage]

Also a note if you want to fix clippy lints later on, it complains a lot about if statements being able to be collapsed, but in actuality I'm doing this do avoid expensive checks like:

if check_for_cycles {
    if self.cycle_detected() {
        self.disconnect(src_node, dst_node, ports_src_dst);
    }
}

if !supports_desired_sample_rate {
    if cpal_config
        .try_with_sample_rate(cpal::SampleRate(sr))
        .is_some() {
    }
}

[slate scarab]

I believe this

if check_for_cycles {
    if self.cycle_detected() {
        self.disconnect(src_node, dst_node, ports_src_dst);
    }
}

is exactly equivalent to

if check_for_cycles && self.cycle_detected() {
    self.disconnect(src_node, dst_node, ports_src_dst);
}

That is, the boolean evaluation should early-out as soon as check_for_cycles is false

[pure dove]

yeah && and || are short-circuiting, so e.g true || panic!() doesn't panic

[dusky mirage]

Oh really? I guess I don't really know how the compiler optimizes it.

[clear wasp]

It's not an optimization, it's part of the language semantics

[pure dove]

this is part of the semantics of almost every language

[dusky mirage]

Oh ok, interesting. TIL

[pure dove]

so that things like x != null && x.property makes sense

or x == null || x.property

in a lot of languages, if you use &/|, those would still end up erroring because the logical operators don't have that property - and operator overloads also don't

but builtin boolean &&/|| always skips

[slate scarab]

clippy's pretty good about not suggesting things that change behavior (well, unless it thinks you ought to change behavior and clearly says so)

[dusky mirage]

Ok, I think I fixed all of the clippy warnings that would cause breaking changes.

@slate scarab Could you do a quick test with the new changes? If nothing is broken, I'll just release the 0.7 version.

You will have to use the new BackendProcessInfo type for your web audio backend.

[slate scarab]

okay my tests pass, the handful of example I ran seemed to work as expected (including the new fancy scheduling stuff) and running foxtrot both natively and on the web works as well

[dusky mirage]

@slate scarab Alright, 0.7.0 is published!

[slate scarab]

thanks a bunch!

[dusky mirage]

Oh yeah, I also had to fix no_std compatibility for firewheel-pool. https://github.com/BillyDM/Firewheel/commit/a010ebe058983e88ffa8e92cc0bcc89b1bd761d1

[dusky mirage]

You're welcome! Exciting to see it getting closer to being included in Bevy!

Darn, missed one thing 😅 . https://github.com/BillyDM/Firewheel/commit/df157cf969a4011388606f8674a4506be6e0e36d

Ok, 0.7.1 is published!

[slate scarab]

oki i finally got another release out #crates message

not perfect, but at this stage it's probably more helpful to publish than sit on it forever

The stream configuration interaction feels like it could be refined, or maybe just more fully abstracted.

[steep dove]

Hey, can yall give me a run-down of what audio stuff is !Send and !Sync and if there are any constraints on what has to be initalized where on different platforms?

[slate scarab]

Firewheel's audio context object is !Send and !Sync. This is not the audio processor -- it's really a bridge between the non-realtime (e.g. game code) and realtime code (i.e. audio processor). We frequently access this context directly from the ECS.

For most platforms, we depend on cpal to handle the lowest level of audio I/O. I'm not an expert in this level of the stack, but I don't believe any of the major desktop environments have hard main-thread requirements for initialization. And, if the platform doesn't require a main thread context, then neither does Firewheel.

There may be some details I'm not aware of though.

I'm very familiar with the web, where the I/O can only be initialized and accessed from the main browser thread. Of course, it would be difficult to do anything else right now in the wider Bevy context, but

[steep dove]

perfect thank you

[oak walrus]

(cc @steep dove ) Platform APIs are not thread safe though IIRC, I'll need to check again but yeah, it didn't matter which thread you call them in, as long as that thread doesn't change for a particular object (it gets really messy because most platforms do expect clients to create their own audio thread, so you need multi threading, but you need to be careful what you call and where.

That being said, I think cpal uses message passing within cpal::Stream, that type might be thread safe

[steep dove]

maybe some of you can review what i've written here https://hackmd.io/@bevy/types_of_nonsend/edit?

[oak walrus]

I'll need to remember to take a look tomorrow but I can so the research and update the doc

[slate scarab]

Since the Firewheel context itself is !Send and !Sync, the thread unsafety of some platform APIs is essentially nullified for our use cases. Any platform APIs would be called from the context's thread, which is necessarily pinned. For me, the main question is whether any platform APIs also require that thread to be the main one.

[steep dove]

just based on the behavior of the other platforms, the suspects would be ios/macos/android. we know web is thread-locked, i'm pretty sure macos is not.

[modest acorn]

Hey all here! Wanting to give a massive thanks to the devs putting hard work into Firewheel and Bevy Seedling. Actually, I've recently decided to abandon a similar project and take up using Firewheel/Seedling instead, given the MUCH better ergonomics and attention they're getting from the Bevy team. For reference, my project is/was MIDI Graph.

There are some differences in design goals, and the biggest one I'm interested in is MIDI file playback. Issue #1 in Firewheel relates to MIDI, but has no description or updates at all. Are there plans to implement a MIDI node that can read in the events from a file a play them through a connected node sub-tree?

[celest whale]

Welcome! This isn't my project per-se, but I'm really happy to hear that you're interested in collaborating

[limpid mason]

The web requirement is interesting. There were some ideas to just run the entire bevy app on a worker thread, do I understand correctly that this won't work then?

[slate scarab]

it could definitely work — you’d just have to do some really annoying coordination with the main browser thread

(I’ve done this before, for reference.)

[oak walrus]

I added a breakdown by OS and platform API, in case that's useful.

[slate scarab]

dang why does asio gotta be like that

[oak walrus]

Steinberg

Well, technically, Steinberg makes no mention of anything and it's up to the driver manufacturers to make thread-safe implementations if they wish. Needless to say, they are not

[slate scarab]

alas

another reason for my distaste for steinberg

(the other being that vsts suck so bad)

[steep dove]

thank you so much

❤️

[oak walrus]

urgh don't talk to me about it, my day job is wrangling with VST3

wrangling with VST3 and Live's abysmal implementation of it

"state-space lifecycle documentation details? that is FUnknown to me"

[viral grove]

I am curious. What is that? 😄

[dusky mirage]

Oh yeah. Firewheel does currently support sending MIDI events from the main thread to plugins. Although there is currently no way for plugins to send MIDI events to other plugins.

I do need to update the project board. Some of those are out of date.

[modest acorn]

Ah, thanks for the info. My previous work was architected such that nodes can send arbitrary events to child nodes, but there were tradeoffs like inflexible timing.
I'll take a dive into Firewheel when I get a chance and see how I can bend it to my needs.

[modest acorn]

Great! An update to the design doc would help newcomers, and perhaps a listing of what node types are implemented might be useful too.

[dusky mirage]

Well shoot, I just hurt my right hand. (I was playing with my nephew when I jumped wrong.) Hopefully it's just bruised and not broken. I might have to type with my left hand for a little while.

[celest whale]

Rest and ice (and medical attention as warranted) 💚

[dusky mirage]

Ok, it's not hurting nearly as bad now, so it's probably just bruised.

[rapid hedge]

Is there a way to find out which sounds are "playing" at a particular translation?

Asking because I'm looking into implementing Thief-like AI, and they mention that they are hooked to the audio engine

So they can ask "Which sounds is this NPC at location xyz hearing? How loud are they?"

[slate scarab]

ya you can just query for SamplePlayers that are actively playing (have a Sampler relationship component) and have a SpatialBasic (or otherwise spatial) effect.

I suppose we could provide a marker component to make it a bit simpler

[rapid hedge]

but that would tell me which samples are audible for the player, not the NPC, no?

or can I use the SpatialBasic to find out how much the sound is dampened for the NPC?

[slate scarab]

oh I see what you mean

[rapid hedge]

(of course, for Thief they also had wayyy more sophisticated 3D spatial audio than what bevy_seedling currently has, but that's another topic)

[slate scarab]

well it depends on how deep you want to go, I suppose

I suppose you'd want to calculate the actual amplitude of sounds from a particular source.

Which is totally feasible.

How many sound sources would you want to track? Would it just be sounds from the player?

[rapid hedge]

Nope. The player can also throw stuff around, for example

[slate scarab]

Oh actually I think you could sidestep calculating it directly in the audio graph.

[rapid hedge]

that would be step 1, but I would also eventually need direction of the sound. Of course, I can just use the Transform of the SamplePlayer as a proxy until we have things like reverb

[slate scarab]

Something that comes to mind is inserting an amplitude meter in the effects chain of all sources that you want to track. That gives you the amplitude of tracked sounds at the source. Then, you could calculate the apparent amplitude in the ECS using the same distance falloff code that the spatial nodes use. This is exposed in Firewheel.

[rapid hedge]

this is from the GDC presentation I'm following

[rapid hedge]

Hmm I see. I would have no clue right now how to implement that in code, but I get the idea

[slate scarab]

it would be quite straightforward! if you're interested, i could make a little demo

[rapid hedge]

That would be really really really helpful 🙂

[slate scarab]

Sound propagation is obviously a lot more work, in the realm of Steam Audio even. I assume they used the sound propagation not just for gameplay logic but also in the actual DSP itself.

[rapid hedge]

yep. Also, this was in the days of hardware-accelerated sound cards 😄

[slate scarab]

Okay maybe it's not actually that complicated for gameplay. You could do some really basic raytracing.

[rapid hedge]

Yeah I expect that it wouldn't be too expensive since there aren't that many gameplay-relevant sounds in a given frame

[slate scarab]

In fact I think you could do basically the same thing as this, but apply the environmental information to the amplitude calculations.

Especially with a little baking, that would be quite cheap. And since you only have one actual listener for the real audio (the player), you could probably skip expensive processing for NPCs.

[rapid hedge]

For a real example, an NPC might shout "Help! There's an intruder!" and that sound will propagate to guards in another room that hear that, alerting them

[slate scarab]

In other words, the NPCs don't need to literally hear the audio. They just need information about it.

[rapid hedge]

bingo

Basically

• Amplitude
• origin
• metadata (e.g. this is a shout for help, or player steps, or a glass breaking)

(As far as I understood their system)

[rapid hedge]

What would you bake?

[slate scarab]

reflection / pathing information

like raytraced GI baking

or maybe it's more like cube maps for reflections

[rapid hedge]

ah I see

[rapid hedge]

Given that we don't have that either in Bevy right now, I think I'll skip on that

(though I am tempted to eventually try to add baking to Solari)

Ah, you may find this interesting

They also use the sound system for smells

since it follows similar propagation rules

This has no effect on the player, but on NPCs

[slate scarab]

it sounds smelly in here

[rapid hedge]

They call it a "pseudo-sound" 😄

Same thing in Half Life

where some NPCs navigate the world by smell

or human NPCs comment on things they smell

[slate scarab]

oh was that one of the creatures in the last episode

or did they do this in the original?

[rapid hedge]

The talk mentions Half Life 1

(since the same dude did the AI there)

And they also use "smells" as a hack to propagate generic data around the room haha

E.g. if something is triggered at a specific location and they want NPCs nearby to react to it by e.g. escaping their prisons

They emit a "smell" for that 😄

Fun stuff

In case this sounds interesting, here's the talk: https://www.youtube.com/watch?v=XF5MLKApWg4

[rapid hedge]

Actually @slate scarab, how would I implement the smell system? I.e. use sound propagation for gameplay purposes but don't actually ever play those sounds

Not even sure SamplePlayer would be correct, since there is not really a "sample" so to speak

I guess I could create some kind of mock sample?

Heh, just use inaudible frequencies

[slate scarab]

ya like presumably the system would provide propagation information between any two points
you could apply this to the actual audio signals, but you could also just use it for simple calculations once per frame

[rapid hedge]

I think they use the audio system for this because it already has this idea of "something spreads between adjacent rooms" builtin

So it makes sense to reuse that

(I know that bevy_seedling doesn't have that yet)

[rapid hedge]

probably could even reuse how real sound goes through walls to emulate how smell goes through e.g. cloth

[slate scarab]

i haven't watched the talk yet, but it seems like the sort of thing that could be done in the userspace of a crate like bevy_seedling -- i.e. you'd calculate the values in the game loop and then send them to the audio system for processing

butttt, you could also use them directly in the game loop for game logic too

but maybe that's what you were already saying haha

[rapid hedge]

(also fun fact: grenades in HL1 emit a smell when they land so that nearby NPCs know that they should jump away from them haha)

[rapid hedge]

I think so 😄

[rapid hedge]

How doable is using seedling / firewheel with audionimbus? The demos are written with cpal, but I suppose I should be able to use firewheel instead. Just asking how much work that would be

[oak walrus]

Instead of having your code in the callback function of the cpal stream, in general, you will have the code in the processing function of a custom node instead

The processing function gives you access to the input and output buffers for the node, you use them the same way you'd use the data out of cpal (though you don't have to do any deinterlacing)

Then it's a matter of registering the node with Firewheel / Seedling, instantiating it, and connecting the node's inputs and outputs

[slate scarab]

ya i haven’t looked into it but it should be quite easy

[rapid hedge]

If I read it correctly, the API gives you deinterlaced results.

[oak walrus]

Firewheel's API, yes

cpal is technically undefined as I think it just forwards the raw buffer from the platform API, but the streams are all configured with interlaced buffers

AFAIK

[rapid hedge]

I meant audionimbus

[oak walrus]

Firewheel's API will give different slices for each channel in the first place

[rapid hedge]

But I see what you mean, thanks 🙂

[rapid hedge]

I was looking into how spatial audio is implemented, and it occured to me that the spatial graph is nearly the same thing as a navmesh

Since I plan on upstreaming rerecast into Bevy eventually, do you think this could be reused for that purpose?

Or is this done mostly with just raycasts instead nowaways?

[slate scarab]

Hm, have you been looking into any particular spatial implementations? The connection isn't immediately obvious to me, but I haven't implemented any sophisticated spatialization myself.

[rapid hedge]

I was just reading a bit into it, but this here seemed interesting: https://www.reddit.com/r/gamedev/comments/109i28i/how_do_games_commonly_do_realistic_audio

There's a link to https://www.youtube.com/watch?v=60P0hzTTJ4Q which I haven't watched yet but would like to later

For me as someone who is just learning about this, it seems like part of the spatialization is doing regular pathfinding?

[slate scarab]

Oh so we're talking occlusion-related simulation

i see the connection now haha

[rapid hedge]

Oops, I see how my terminology was inappropriate

[slate scarab]

well things like occlusion fall under the spatialization umbrella, it's kind of a broad term i think

You could probably get some decent occlusion simulation with navmeshes. Obviously you wouldn't catch everything (like an open window you can't go through), but it would probably be pretty fast

although it seems like it would be tricky to gather surface information along the path -- in other words, what kinds of materials it would hit, which have different absorption and reflection properties

[rapid hedge]

Pathfinding usually has the concept of "zones" with different traversal costs

Maybe that could be used?

And rerecast allows arbitrary user data for each navmesh polygon

But yeah it's not exactly "reflection"

I guess you could say "this room is primarily made of stone, so when the sound goes through it, treat it as if it bounced off of stone"

Maybe "treat it as one bounce every n meters"

(idk I have no experience with this 😄)

[slate scarab]

mm it might work, to a close enough approximation
you'd probably want a bit of additional information though, like whether the source is occluded at all, maybe with some direct ray casts

[rapid hedge]

gotcha

I mean, I probably will use Steam Audio anyways

was just thinking that this may be something to keep in mind for an upstream solution

[slate scarab]

i imagine we'd probably just have third-party crates to do this sort of thing, unless we really went for it like the steam audio port

it feels like there'd be too many compromises or approximations otherwise

[oak walrus]

Oh yeah that makes sense, you could be taking the length of the path to compute the filter cutoff for air absorption, and also delaying the sound for propagation, and panning by using the direction of the final segment of the path for panning, that would probably work decently enough for close-quarters or outside scenes

Though that would mean having a 3d navmesh to account for sound waves going above buildings

And you'd probably want multiple paths returned to get a weighted average signal path

[slate scarab]

I believe the shortest path is often used as a decent approximation, even by sophisticated approaches.

[oak walrus]

It is an approximation but everything is approximations and workarounds in audio, so 🤷

[slate scarab]

I think steam audio only does multiple path simulation with baked path information, but I could be wrong

[dusky mirage]

Firewheel's built in spatialization node includes a lowpass filter for muffling sounds. It's of course not nearly as fancy, but it's really cheap on the CPU.

[slate scarab]

basic occlusion is certainly very easy -- just do a raycast, see how much stuff it has to travel through, and then slap some extra low-pass + attenuation on that bad boy

I think some games actually just do that

[dusky mirage]

It would be nice to have a convolutional reverb node to simulate spaces. (Though convolution is pretty expensive on the CPU, so it would be more of a global effect).

I think it's pretty common for example to turn on a global reverb when the player enters a cave and such.

[slate scarab]

half life 2 had a fancy system where they'd gather information about the level and basically generate reverb parameters on the fly, but of course they only had one "volume" happening at a time

[dusky mirage]

And there is of course the FreeVerb algorithmic reverb, but the sound quality isn't that great.

[oak walrus]

We can do GPU FFT for the convolution, ez

[slate scarab]

well theoretically, with enough latency...

[oak walrus]

But other than that, a simple delay-based feedback reverb with a diffusion tank should be cheap enough

[oak walrus]

That's basically all GPU Audio can do really at the moment, so yeah, definitely

[dusky mirage]

I wish CPUs had built in FFT accelerators.

[slate scarab]

seems to be relegated to specialty silicon or FPGAs
i guess it's hard to make something general enough without dedicating crazy amounts of die space

[dusky mirage]

Though a lot of die space is already dedicated to video decoding/encoding. I wish audio would get the same treatment.

(And nowadays a lot of die space is dedicated to AI accelerators.)

[oak walrus]

No idea we had if statements accelerators now

[slate scarab]

clearly processor manufacturers don't want us to have any fun

[dusky mirage]

Ah yeah, speculative execution.

I'm surprised modern processors don't have hardware accelerated trig and square root functions. But I'm not a computer hardware engineer, so there may be a good reason for that.

Some ARM CPUs do have hardware accelerated matrix multiplication which is nice.

(And of course GPUs have hardware accelerated matrix multiplications galore)

Why no hardware accelerated ECS?

[slate scarab]

transistors as entities

[oak walrus]

circuits as relationships

[dusky mirage]

I wouldn't be surprised if CPUs started getting JavaScript accelerators.

And DOM accelerators.

[slate scarab]

maybe one day we'll get stack machine accelerators, like for Wasm

[normal python]

WASM-32 architecture is the future

[rapid hedge]

@formal nexus can landmass return multiple alternate paths?

[formal nexus]

Not currently. I mean I can make it do that but that problem doesn't seem well defined? Like an alternate path could be just taking one step off the optimal path and then another right back onto the optimal path. That doesn't really add any extra information?

[slate scarab]

i think there's no way around some kind of path tracing for proper approximations (which can be partly baked as steam audio does)

[clear meadow]

The reverb in fundsp sounds pretty good. Modern CPUs can handle audio DSP easily.

Path tracing could be done easily in 3D using a convex decomposition of open spaces. From what I understand, a navmesh is a 2D version of this.

[formal nexus]

Yup, that's accurate!

[oak walrus]

If you're in the open, I guess it doesn't really matter, you take the shortest path and use that, I was thinking in case of an inside scene where the sound source is outside, and could come through windows or open doors

in which case even in real life you'd hear it coming from both

(and if the path is sufficiently long, a slight delay between them)

[clear meadow]

It might make sense to cast rays around the listener, then trace paths from each of those points? Sound waves behaves somewhat like rays, at least at higher frequencies. That's another possibility, to use different techniques for different frequencies.

[oak walrus]

It starts becoming too much I think. Multiple raycasts and multiple navmesh queries every frame, for each sound source, is a lot of budget for the audio system

Frankly even with a single navgraph query per source for getting distance and direction of the sound would already give something nice I think

And again, there's a lot of faking, slap a reverb volume and it sounds immersive

[clear meadow]

I think the trick would be to mark particular sounds as high frequency short sound effects, and only spatialize those sounds. Low frequency and ambient sounds could use a simpler approach.

[oak walrus]

It doesn't make sense really, almost every sound is full band, and consistency is more important than trying to shoehorn ray tracing

[drowsy dome]

this would be fun to build

[vocal dock]

0Hz, the best inaudible frequency

[slate scarab]

some NES games would set the triangle channel really high as a way to "turn it off"

[rapid hedge]

No rush, just wondering: how far away is a 0.17-rc release?

I'm looking at my dependencies and judging which I can migrate myself, and I don't think I'm up to the task of migrating seedling myself haha

And plenty of my deps like hanabi can just be commented out, the game will run fine without VFX for a moment

But seedling for better or worse is quite "intrusive" in its integration, i.e. going back to bevy_audio is not as easy as just commenting out a system

Which is fine btw, I wouldn't want to change the API to mitigate this

but it also means that I now also need to wait for seedling before I can use 0.17

But again, no rush

Y'all are by far not the only blockers now and the RC just released

[slate scarab]

Hm, is the convention to release an rc version on crates.io for ecosystem crates? I've never actually participated in an rc before.

[rapid hedge]

yep 🙂

You basically follow the naming convention of Bevy

So now you'd release a bevy_seedling at 0.6.0-rc.1

And usually you don't need to update the RC

[slate scarab]

ya that would be super easy (i think @drowsy dome has already done pretty much everything), although we'll need to get one in for Firewheel as well

[rapid hedge]

Since we try to not have breaking changes from one RC to the next

[rapid hedge]

great!

[slate scarab]

I've been meaning to get a patch release out for a while, and this would certainly be a good reason to just do it haha.

@dusky mirage if you don't mind, I'd love to help get out an rc version of Firewheel. Should be extremely minimal, and I can verify those Component derives like you mentioned a few weeks ago.

[rapid hedge]

thanks for looking into it

appreciate it!

[dusky mirage]

Ok, cool. Let me know what you need!

[slate scarab]

we should be able to get everything on the bevy_seedling side sorted later today 👍

[drowsy dome]

@dusky mirage I've got a firewheel 0.17-rc1 branch: https://github.com/BillyDM/Firewheel/compare/main...dsgallups:Firewheel:0.17-rc1

Updated branches:
firewheel: https://github.com/dsgallups/Firewheel/tree/0.17-rc1
bevy_seedling: https://github.com/dsgallups/bevy_seedling/tree/0.17-rc1
firewheel-ircam-hrtf: https://github.com/dsgallups/firewheel-ircam-hrtf/tree/0.17-rc1

feel free to open PRs into your crates if you'd like. I'm not sure how y'all wanna go about structuring your branches for release candidates, so I've not opened these to target your default branches

[dusky mirage]

Since we are publishing a new release, there are a few minor breaking changes I want to make to Firewheel and release a 0.8 version.

[drowsy dome]

fwiw, the way Jan has updated his crates was with his own suffix. i.e. https://github.com/janhohenheim/bevy_kill_my_cache is now 0.3.0-rc.0 (I imagine this crate probably won't have any breaking changes 😆 )

[dusky mirage]

The pull request for Firewheel looks good to me if it looks good to you @slate scarab

[slate scarab]

ya i thought it looks good
i do wanna test everything before pushing go, which i should be able to do in like an hour actually

[dusky mirage]

You mean test the Firewheel changes, or your bevy-seedling changes?

[slate scarab]

I think the Firewheel changes are probably fine, but I wouldn't mind making sure everything integrates properly together. but there's not too much that can go wrong

[drowsy dome]

oh btw, I didn't remove the glam-29 feature, though I think it might be kinda dead unless there are dependents specifically on 0.29. I don't believe that version will be maintained in the future

just following the minimum change principle

[dusky mirage]

Hmm, that is true. I'm not sure we really need to have the glam-29 feature.

[slate scarab]

you could keep around older versions for a few Firewheel versions, since bevy projects aren't the only dependents of Firewheel (although bevy projects are the biggest atm)

[dusky mirage]

@slate scarab Ok, the 0.8 branch is ready for testing! There is one small breaking change you will probably need to add for the hrtf node (you will need to add a coeff_update_factor parameter like what has been done in the SpatialBasic node).

https://github.com/BillyDM/Firewheel/tree/0.8

Also a side note (and this won't matter as much once we get a stable version published), I need to decide sooner or later whether I want to move Firewheel to Codeberg or not. I've been thinking of moving my other projects there. But I'm not sure if Firewheel being closely tied to Bevy will add too much friction or not.

[slate scarab]

i'm sure @celest whale would have some insightful thoughts about that

dang we finally have to deal with getrandom 0.3

[dusky mirage]

Oh yeah, and @drowsy dome, if I do decide to move my projects away from GitHub, would you be ok with switching to Liberapay or OpenCollective? GitHub sponsors has hardly done anything for me anyway.

I do have a liberapay, but OpenCollective looks a lot more polished and user friendly.

[drowsy dome]

np

[dusky mirage]

Cool!

I'll try applying for OpenCollective first. If for some reason they don't accept my projects, then I'll stick with Liberapay.

[slate scarab]

btw with the rc process, we'll want another breaking change once the rc period completes and bevy 0.17 is released
that's why @drowsy dome added the -rc.1 suffic
if you're chill with bumping to 0.9 then it's no problem either way

[dusky mirage]

Oh, are you saying I should add the -rc.1 suffix to the Firewheel crates too?

[drowsy dome]

probably before 0.17 is officially released so that users have to explicitly set that version if that makes sense. though it probably only applies to people with firewheel = "0.*". In that instance, they'd grab your release even if it's "not in final form"

that probably wouldn't be a common case though

but people with catch alls will not grab suffixed versions

(that's part of why I set bevy_platform from 0.* to 0.17.0-rc1. Because the former wouldn't update)

[slate scarab]

You certainly don't have to -- just keep in mind that exiting the rc phase will call for another breaking change. If it feels like a lot of churn, we might be able to smooth the process out in the future

Although... can you publish beta versions to crates io with github dependencies?

[drowsy dome]

yes

[slate scarab]

If that were the case then we wouldn't be forced to make a Firewheel release.

[drowsy dome]

you can use [patch.] in the toml to point to specific branches on your release

so firewheel doesn't have to publish any release

[slate scarab]

And crates io is chill with that?

[drowsy dome]

yes

[slate scarab]

Hm, well in that case no rush @dusky mirage haha

[drowsy dome]

i.e.

[patch.crates-io]
firewheel = { git = "https://github.com/dsgallups/Firewheel.git", branch = "0.17-rc1"}

the only thing that must remain the same is the version number

[slate scarab]

right ya i just assumed crates io wouldn't let you publish it

[drowsy dome]

so if Billy bumps the version in the branch, but doesn't release, outta luck

[drowsy dome]

oh btw, to be explicit, probably a good idea to pin to a commit, not the branch

[slate scarab]

ya definitely

that would avoid a catastrophic failure haha

[rapid hedge]

the newest Bevy CLI alpha off of main does it for you FYI 🙂

[slate scarab]

oh it would be nice if i could binstall it for ci

[rapid hedge]

- name: Install Bevy CLI
  run: cargo binstall --locked --no-confirm --force --git='https://github.com/TheBevyFlock/bevy_cli' bevy_cli

[slate scarab]

ya although i guess for now it'll just have to compile it
idk how binstall works but im guessing it doesn't store unpublished bins

[rapid hedge]

@limpid mason does the Bevy CLI GH action allow using the CLI off of main?

[rapid hedge]

AFAIK it only looks at releases, yes

[slate scarab]

oh there's an action?

now we're cooking

[rapid hedge]

Yeah but I don't use it since I'm a perpetual user of the newest Bevy CLI features in main haha

case in point, the getrandom stuff

[slate scarab]

Hm, no looks like we can't get away with not publishing anything. @dusky mirage looks like we'll need a new published version if we want to adhere to the rc! Sorry for the trouble. Feel free to version it 0.8.0-rc.1 if you want to publish a proper 0.8.0 after the rc.

0.8 seems good from my end, the adjustment to ircam-hrtf was pretty simple.

[dusky mirage]

Alright, I made the version 0.8.0-rc.1! Do you need me to publish it to crates.io, or do you only need that for the full 0.8.0 version?

[slate scarab]

ya unfortunately (or fortunately?) all dependencies of a crate published to crates.io must also be published

[dusky mirage]

Ok, I'll do that real quick then!

Alright, published!

[slate scarab]

@rapid hedge is foxtrot (mostly) updated for the rc? I want to sanity test everything in a real project before publishing (and mine aren't updated for the rc atm). Let me know if there's an in-progress branch or anything!

[rapid hedge]

Not yet, I'm currently on a PR quest to port everything necessary 😄

I'm nearly done though

[drowsy dome]

ya I was totally wrong. I forgot that y'all don't include your Cargo.lock s, so everything I said was wrong

so my bad 😅 it would be true if the Cargo.lock were part of your crates. I always include my lockfile so I can get away with depending on patches

but yeah makes sense

[slate scarab]

that's okay, i really appreciate the changes regardless!

[limpid mason]

Honestly cant remember, perhaps @frozen zephyr or @hearty bear know?

[frozen zephyr]

There actually isn't a CLI Github Action

There's only one for the linter right now

You can cargo-binstall the CLI, but it only supports the latest release (not main)

We can definitely add a Github Action to install the CLI in the future, but that probably will be done after v0.1.0-alpha.2 is released

[frozen zephyr]

-# And we use a funny hack for this due to a cargo-binstall bug

[rapid hedge]

Oh whoops I got them confused

[rapid hedge]

Update: I'm pretty sure I'll have something working by tomorrow where you can test bevy_seedling for the RC 🙂

[slate scarab]

I'm actually feeling pretty confident about it at this point. My test suite and examples should be decent enough at least. I'm basically ready to publish, but I'm getting a weird linker error in CI for wasm

But I wouldn't mind waiting a day. My versioning would be all off if I had to publish a fix 😅

If you don't end up getting it going tomorrow, no worries! I can just publish it at that point.

[drowsy dome]

anecdotally, everything works for me, which is two custom nodes, sampler node with some sampler effects, it's really

[rapid hedge]

Oh if everything works you don't need to wait for me 😄

[drowsy dome]

wondering if there's a system you'd like to test the edges of bevy_seedling, or if it's not there yet :o

one of the things I was thinking about going through the examples was building a snapshot system to test timed output samples between versions and capturing any diffs

[slate scarab]

i think rigorous testing like that, especially when examining audio data, is pretty tricky

but it would be cool to have

huh, well something's up with the runner, not sure what's causing the failure
building it locally works

[drowsy dome]

I've got a system that will compare my sf2 synth's output to rustysynth at least, and I use it to compare versions of my synth whenever i make changes

so maybe it's node-specific, but could be cool to do something a bit more agnostic

I'm going to open source the sf2 synth in a week or so anyway. will see after that

[slate scarab]

now if only we could solve those pipewire issues
seems to be related to https://github.com/RustAudio/cpal/issues/605

which is uh, not promising

[drowsy dome]

working on it

but why is this new

that's what I'm wondering

I need to definitely go back to an older version and see if I get this issue. if I do, then 🤷‍♂️

[drowsy dome]

moving to seedling chat

[rapid hedge]

Whew, upgrading to feathers UI took a bit longer than I thoughts

lots of quirks

So I didn't check out bevy_seedling 0.6.0-rc yet

but I started porting my binaries now 🙂

[slate scarab]

okay bevy_seedling 0.6.0-rc.1 is up :)

[rapid hedge]

@slate scarab what do I migrate this to?

// right now, `Default` isn't implemented for any non-cpal backend
#[cfg(feature = "web")]
app.add_plugins(
    bevy_seedling::SeedlingPlugin::<firewheel_web_audio::WebAudioBackend> {
        config: Default::default(),
        stream_config: Default::default(),
        spawn_default_pool: true,
        pool_size: 4..=32,
    },
);

sorry if you already wrote that down somewhere

I admit I didn't check your migration guide(s) at all 😄

[slate scarab]

oh i missed this in the guide 😅

[features]
web = ["bevy_seedling/web_audio"]

[dependencies]
bevy_seedling = { version = "0.6.0.rc-1" }

#[cfg(feature = "web")]
app.add_plugins(
    bevy_seedling::SeedlingPlugin::new_web_audio(),
);

https://docs.rs/bevy_seedling/latest/bevy_seedling/struct.SeedlingPlugin.html#method.new_web_audio

You no longer need to depend on firewheel-web-audio directly if you don't need anything else from it.

[rapid hedge]

Much better!

[rapid hedge]

Looks like firewheel is panicking on webgpu for me

Also, yep, 0.2.101 is the latest wasm-bindgen supported by seedling

[slate scarab]

Oh, is there an issue past that? Is it seedling in particular causing an issue?

[slate scarab]

That seems odd. I wonder how a negative value could be calculated here. Seems almost like a chrome bug.

Or it could be unrelated to webgpu -- just a coincidence, with some likelihood of occurring at any point.

[rapid hedge]

#1404237363550486751 message

I think it's seedling

but I'll have to double check

it's certainly not bevy itself

[slate scarab]

What happens if you compile with atomics without seedling?

[rapid hedge]

let me check

[slate scarab]

My recent wasm-bindgen downgrade was actually in a project without any audio at all (well, not any high-level audio anyway).

In other words, I encountered the same error, without seedling but with multithreading.

[rapid hedge]

I see 👀

[slate scarab]

I found an issue for it -- I think some people had the same issue. Let me find that.

well... https://github.com/wasm-bindgen/wasm-bindgen/issues/3315#issuecomment-3301868064

not that great 😅 but I see others have had success with 0.2.101

__wasm_call_ctors, huh?
could that be related to reflect autoregistration?

[rapid hedge]

good idea

it's weirdly enough another error than before

the heck

I cannot compile a fresh project at all using atomics + webgpu

but I can compile foxtrot

(on that specific wasm-bindgen version)

[slate scarab]

oh man

experimental rust features + experimental chrome features = sometimes working binary???

[rapid hedge]

y'know what

rustup update + cargo clean

let's see now

sudo dnf update google-chrome too

and rm -r ~/.cargo/registry/src/*

the nuclear option

oh hey good idea let me ju

[slate scarab]

it feels too dangerous to be kept alive ngl
some things are better left well alone

[rapid hedge]

arrrg

okay

let's try without atomics

[hearty bear]

is your font rendering a bit bonked?

[rapid hedge]

uuuh not that I know of?

(I completely forgot which font I use lol)

[hearty bear]

maybe its also just the terminal emulator (dont look to close or you cant unsee it )

[rapid hedge]

Is... is the bevy cli broken?

[rapid hedge]

send zoomed in screenshot plz 😄

I'm just deleting line after line of config at this point

[hearty bear]

the r gets cut off a bit and some letters are a bit blurry imo

[rapid hedge]

Am... am I blind??

Well I know my eyesight is shit haha

[hearty bear]

well mine too

[rapid hedge]

Okay Albrecht

Since you're here

[hearty bear]

NOOOO

[rapid hedge]

This is my Cargo.toml

[package]
name = "test"
version = "0.1.0"
edition = "2024"

[dependencies]
bevy = "0.17.0-rc"
wasm-bindgen = "=0.2.101"

This is my main.rs

fn main() {}

[hearty bear]

yesssir

[rapid hedge]

my bevy cli is just updated

bevy 0.1.0-alpha.2

And I have renamed my ~/.cargo/config.toml to _config.toml

[hearty bear]

do you need the wasm-bindgen dependency?

[rapid hedge]

so it should be off

[rapid hedge]

well, not right now

but on foxtrot, the newest wasm-bindgen didn't work

but let me remove it for this example, you're right

sec

removed

then ran cargo update

followed by bevy run web

And I get

hhh@hhh-fedora ~/g/test (main) [SIGINT]> bevy run web
info: automatically configuring `getrandom` web backend
info: compiling to WebAssembly...
   Compiling test v0.1.0 (/home/hhh/git/test)
    Finished `web` profile [unoptimized + debuginfo] target(s) in 0.17s
info: bundling JavaScript bindings...
error: failed to find intrinsics to enable `clone_ref` function
warning: failed to run wasm-bindgen, trying to find automatic fix...
error: command `wasm-bindgen --no-typescript --out-name test --out-dir /home/hhh/git/test/target/wasm32-unknown-unknown/web --target web /home/hhh/git/test/target/wasm32-unknown-unknown/web/test.wasm` exited with status code exit status: 1

I'm on nightly, maybe that's why

[hearty bear]

oh yea

i legit dont know what that is

[rapid hedge]

@slate scarab let me know if this is getting too off-topic for this channel

I'm mainly posting here because it's keeping me from using the multi-threaded audio backend

[slate scarab]

As the sole moderator and authority of this channel, I deem it perfectly cromulent

But uh, idk I think it's fine!

Let me give this a try real quick too

[rapid hedge]

TIL

appreciate any and all Simpsons references

Oh hey Alice

[celest whale]

🤨 || I'll allow it||

[rapid hedge]

stable doesn't work either

am I cursed?

[steep dove]

what are you trying to do?

[limpid mason]

I think that refers to a Wasm feature that's not enabled or enabled when it shoukdt be?

[rapid hedge]

bevy run web on an empty Bevy project

[limpid mason]

Can you run with --verbose?

[steep dove]

multi-threaded audio backend
i assume this uses atomics, which means you need to rebuild the stdlib.

[limpid mason]

That's handled automatically by the CLI

[rapid hedge]

hhh@hhh-fedora ~/g/test (main) [1]> bevy run web --verbose
debug: running: `cargo metadata --format-version 1`
info: automatically configuring `getrandom` web backend
info: compiling to WebAssembly...
debug: running: `cargo build --config profile.web.inherits="dev" --config profile.web-release.inherits="release" --config profile.web-release.strip="debuginfo" --config profile.web-release.opt-level="s" --bin test --profile web --target wasm32-unknown-unknown`
debug: with env: RUSTFLAGS= --cfg getrandom_backend="wasm_js"
    Finished `web` profile [unoptimized + debuginfo] target(s) in 0.12s
info: bundling JavaScript bindings...
debug: running: `wasm-bindgen --no-typescript --out-name test --out-dir /home/hhh/git/test/target/wasm32-unknown-unknown/web --target web /home/hhh/git/test/target/wasm32-unknown-unknown/web/test.wasm`
error: failed to find intrinsics to enable `clone_ref` function
warning: failed to run wasm-bindgen, trying to find automatic fix...
debug: running: `wasm-bindgen --version`
error: command `wasm-bindgen --no-typescript --out-name test --out-dir /home/hhh/git/test/target/wasm32-unknown-unknown/web --target web /home/hhh/git/test/target/wasm32-unknown-unknown/web/test.wasm` exited with status code exit status: 1
hhh@hhh-fedora ~/g/test (main) [1]>

[rapid hedge]

I disabled all of that

[steep dove]

[rapid hedge]

this should be a blank setup

If I didn't fuck up

[rapid hedge]

now in color

[rapid hedge]

that's a damn good emoji I must say

look at the subtle off-white coloring

[limpid mason]

Did you try updating wasm-bindgen? Or did you just recently update it?

[rapid hedge]

The Bevy CLI just installed it for me, yes

[steep dove]

is it this https://github.com/trunk-rs/trunk/issues/951

are you on a mac?

[rapid hedge]

Fedora 42

[rapid hedge]

I can install it manually though

[rapid hedge]

But that shouldn't be necessary:

hhh@hhh-fedora ~/g/test (main) [1]> wasm-bindgen --version
wasm-bindgen 0.2.104

[slate scarab]

apparently some other folks here have seen that failure though

[hearty bear]

yea i got this error from wasm-bindgen too on empty projects but the moment you actually do something it should go away

like if you include the bevy prelude for example it should already run?

[rapid hedge]

Oh okay, let's do that

YESS that works

Okay

now let's try adding atomics back in

step by step

[slate scarab]

huh

[rapid hedge]

dear Rust-Analyzer
I disagree with the notion that this is "unused"
apparently just importing it fixes things

Okay bevy run web -U multi-threading runs

now let's add seedling

[hearty bear]

its the coconut image of bevy

[limpid mason]

That's so weird lol
Is it caused by some change in bevy or some new version of wasm-bindgen?

I don't think I've run into this before

[rapid hedge]

version = "0.6.0-rc"*

[slate scarab]

Sorry 😅 I had a feeling in the back of my head this whole time that version string was wrong.

[rapid hedge]

well it seems like seedling is running

[hearty bear]

nono its a wasm-bindgen issue that has been around for some time i think (https://github.com/trunk-rs/trunk/issues/951#issuecomment-2870258993)

[rapid hedge]

is that good

@hearty bear I don't need anything else than -U multi-threading right?

Like, no manual header setup etc.

or explicitly enabled atomics

[slate scarab]

well, uh... i wouldn't call it ideal
that would only happen if it failed to initialize the JS shim

which should be unrelated to cross origin isolation

[rapid hedge]

maybe caching, sec

nope

still here

for reference, here are my files

[slate scarab]

oh no it's panicking in a context without a text decoder

so the message is being hidden

[rapid hedge]

[package]
name = "test"
version = "0.1.0"
edition = "2024"

[dependencies]
bevy = { version = "0.17.0-rc.1", default-features = false, features = [
  "std",
  "async_executor",
  "android-game-activity",
  "android_shared_stdcxx",
  "animation",
  "bevy_asset",
  "bevy_color",
  "bevy_core_pipeline",
  "bevy_post_process",
  "bevy_anti_alias",
  "bevy_gilrs",
  "bevy_gizmos",
  "bevy_gltf",
  "bevy_input_focus",
  "bevy_log",
  "bevy_mesh_picking_backend",
  "bevy_pbr",
  "bevy_picking",
  "bevy_render",
  "bevy_scene",
  "bevy_image",
  "bevy_mesh",
  "bevy_camera",
  "bevy_light",
  "bevy_shader",
  "bevy_sprite",
  "bevy_sprite_picking_backend",
  "bevy_sprite_render",
  "bevy_state",
  "bevy_text",
  "bevy_ui",
  "bevy_ui_picking_backend",
  "bevy_ui_render",
  "bevy_window",
  "bevy_winit",
  "custom_cursor",
  "default_font",
  "hdr",
  "ktx2",
  "multi_threaded",
  "png",
  "reflect_auto_register",
  "smaa_luts",
  "sysinfo_plugin",
  "tonemapping_luts",
  "webgl2",
  "x11",
  "wayland",
  "debug",
  "zstd_rust",
] }
bevy_seedling = { version = "0.6.0-rc", features = ["web_audio"]}

This is from the seedling docs

and here's my main

use bevy::prelude::*;
use bevy_seedling::prelude::*;

fn main() {
    App::default()
        .add_plugins((DefaultPlugins, SeedlingPlugin::new_web_audio()))
        .run();
}

am I missing anything?

[hearty bear]

are you on nightly again? @rapid hedge

[limpid mason]

Yes that should work

[rapid hedge]

yep

on the newest nightly

[limpid mason]

Jan is always on the bleeding edge :D

[hearty bear]

ye was just asking since he switched to test the wasm-bindgen ^^

[slate scarab]

testing now

[rapid hedge]

thx

[rapid hedge]

I'm just running bevy run web -U multi-threading

and opening in chrome

which I start with google-chrome --enable-features=Vulkan

[hushed widget]

stable? oh you mean yesterday nightly? 😄

[rapid hedge]

hehehehe

[slate scarab]

oh man, i've been spoiled with my bevy_ecs projects lately
all this rendering stuff takes a while 😅

What wasm bindgen are you on?

[hearty bear]

i think this works for me?

[rapid hedge]

also newest nightly?

[hearty bear]

i always use the nightly version that the linter uses so im on nightly-2025-08-07

[rapid hedge]

And on chrome Version 140.0.7339.207 (Official Build) (64-bit)

[rapid hedge]

let me use that

[slate scarab]

I ran into the unknown instruction failure immediately with 0.2.104 on macos.

[rapid hedge]

newest

which is weird

[hearty bear]

oke i will get my mac just for you..

[rapid hedge]

given that I couldn't compile it on that before

[rapid hedge]

[rapid hedge]

I get the same error there

[slate scarab]

oh the cli is so polite about tools

[rapid hedge]

I love that feature

@slate scarab what's the simplest example I can try on the bevy_seedling repo?

[slate scarab]

Okay so without running chrome explicitly with webgpu support, it works great (if I pin wasm-bindgen to 0.2.101).

[rapid hedge]

ha!

[slate scarab]

use bevy::prelude::*;
use bevy_seedling::prelude::*;

fn main() {
    App::default()
        .add_plugins((DefaultPlugins, SeedlingPlugin::new_web_audio()))
        .add_systems(
            Startup,
            |mut commands: Commands, server: Res<AssetServer>| {
                commands.spawn(SamplePlayer::new(server.load("any-file.wav")));
            },
        )
        .run();
}

But on the repo idk one_shot probably

basically just this ^

actually no i think i am running with webgpu support

[rapid hedge]

I thought maybe Firefox would yield more success

[hearty bear]

this is just your nightly report 👍 i tried on my linux machine with firefox and i saw no errors

[slate scarab]

Okay yeah even explicitly launching chrome with that flag seems to work.

[hearty bear]

but on my mac on chrome i get the same error (v 140.0.7339.208)

[rapid hedge]

yay I'm not alone

[rapid hedge]

And you used -U multi-threading?

[slate scarab]

ya I wouldn't have audio without it!

[rapid hedge]

[slate scarab]

Hm, I'm on 140.0.7339.134, to be clear

[rapid hedge]

very cool

I hope this doesn't end with me booting my Windows

[slate scarab]

ohhh also not on latest nightly

[rapid hedge]

I tried the one_shot example by modifying it to use the new audio backend

wake up Corvus, new error just dropped

[slate scarab]

wtf

who's trying to unwind

[rapid hedge]

it's this code:

//! This example demonstrates how to play a one-shot sample.

use bevy::{log::LogPlugin, prelude::*};
use bevy_seedling::prelude::*;

fn main() {
    App::new()
        .add_plugins((
            MinimalPlugins,
            LogPlugin::default(),
            AssetPlugin::default(),
            SeedlingPlugin::new_web_audio(),
        ))
        .add_systems(
            Startup,
            |server: Res<AssetServer>, mut commands: Commands| {
                // Spawning a `SamplePlayer` component will play a sample
                // once as soon as it's loaded. If no pool is specified
                // and no effects are applied, the sample will be played in
                // the `DefaultPool`.
                commands.spawn(SamplePlayer::new(server.load("caw.ogg")));
            },
        )
        .run();
}

running with bevy run --example one_shot --features web_audio web -U multi-threading

[rapid hedge]

which nightly are you on?

I want to try that

[slate scarab]

2025-07-05, but I'm updating now

[limpid mason]

I might as well also try it

[hearty bear]

this works for me on chrome

i got a bit scared by the sound

[slate scarab]

CAW

[rapid hedge]

CAW

[hearty bear]

CAW

[slate scarab]

Looks like the latest nightly also works for me

[limpid mason]

So I need the bevy_seedling example, chrome and nightly rust to reproduce?

[rapid hedge]

and my machine, apparently