V12 Audio API Changes

[fiery kiln]

Join the thread to discuss!

[wooden loom]

Sequencer is using game.audio.play() and the Sound#stop method as well as its stop and end events, that's pretty much it 🙂
https://github.com/fantasycalendar/FoundryVTT-Sequencer/blob/master/src/modules/sequencer-audio-helper.js#L41-L45
It does not change the behavior of sounds, merely start/stop them.

[fiery kiln]

great, you're in the clear!

[wooden loom]

I also do adjust the volume in real time, but that seems containable within your parameters

[zenith cargo]

Mostly out of curiosity (and I suspect the answer is no)

Are the AudioHelper methods impacted at all?

[fiery kiln]

Yep, the Sound#volume and Sound#fade methods work much the same as before, although the technical implementation of Sound#fade is improved.

[wooden loom]

Excellent, that helps more than it hinders

[fiery kiln]

There are some AudioHelper changes, documented in the issue involving caching changes, otherwise no.

@wooden loom Glancing over your code briefly I see a few things that could be improved once you are ready for the module to require V12 (which understandably might not be for some time).

For example:

    if (data.duration) {
      setTimeout(() => {
        sound.stop();
      }, data.duration);
    }

Could be:

sound.stop({delay: data.duration});

Which now internally uses the new AudioTimeout which is a special timing mechanism for very precise audio playback timing control.

Basically anywhere you use window.setTimeout for anything related to sound playback you'll be better off using AudioTimeout instead.

[vital socket]

Light Mask is likely not affected. It messes with the shape of the sound placeable, but not the underlying sound api.

[dull iron]

I'll have to check, but I believe Soundscape does not use the Foundry audio api, but it does everything through the web audio api. Which I assume is not affected.

[hot zealot]

Thanks for the ping, yes I am only setting the MAX_BUFFER_DURATION!

[fiery kiln]

Awesome. You'll be unaffected in V12 although there will be a compatibility warning and by the time you get to V14 you'll need to switch from AudioContainer.MAX_BUFFER_DURATION to foundry.audio.Sound.MAX_BUFFER_DURATION.

[fiery kiln]

Super, of course use of the native web audio API will be unaffected. The new V12 Sound API is a really nicely designed (in my biased opinion) layer on top of the web audio API which allows you to do some powerful things via an easier interface. Things like:

• Optimized LRU cache for buffers
• Ability to pause/stop/replay/seek audio buffers without having to deal with recreating the source node
• Scheduled timings/transitions that use precise audio context timing
• Managed volume transitions and fades
• Ability to mix large/long files (via media streaming nodes) and short buffered files (via audio buffer) seamlessly without worrying about when to use which

If that sounds appealing you might check out the new API if you find yourself revisiting Soundscape in V12, I don't think you'll have any forced changes though if you're just using native JS stuff.

[vague umbra]

Oh wow, something that actually might break one of my mods. I've been riding the gravy train for 5 releases now 😦

[fiery kiln]

Which one do you think is probably impacted, do you want me to give it a look over?

[flint cape]

@fiery kiln in terms of premium modules, i have Dynamic Soundscapes but it only interacts with high leevel methods (playlist\sound methods and document updates)
Then i have 3D canvas which has an option to use 3d positional audio (basically sets the volume of a canvas sound based on it's distance from the camera)

this might be the more suspicios sound related code i have ```js
getSoundFrequency() {
const sound = Array.from(game.audio.playing.values())[0];
if (sound && sound == this._sound && this._analyser) {
this._analyser.node.getByteFrequencyData(this._analyser.data);
let bass = 0,
mid = 0;
const length = this._analyser.data.length;
for (let i = 0; i < length; i++) {
if (i < 20) {
bass += this._analyser.data[i];
} else if (i < 40) {
mid += this._analyser.data[i];
}
}
bass /= 20;
mid /= 20;
//return new THREE.Vector3(1 + mid/255,1 + bass/255,1 + mid/255);
return new THREE.Vector3(1 + bass / 255, 1 + mid / 255, 1 + bass / 255);
//return new THREE.Vector3(1 + bass/255,1 + mid/255,1 + treble/255);
}
this._sound = sound;
if (!sound) return new THREE.Vector3(1, 1, 1);
const {
container: { sourceNode },
context,
id,
} = sound;
const analyserNode = new AnalyserNode(context, { fftSize: 4096 });
sourceNode.connect(analyserNode);
this._analyser = {
node: analyserNode,
data: new Uint8Array(40),
};
return new THREE.Vector3(1, 1, 1);
}

If it's a chance to get some extra APIs, i have a massive Override for SoundsLayer#refresh for no reason. I have to drill very deep just to add this condition here

                  // Determine whether the sound is audible, and its greatest audible volume
                    if (useCameraDist) {
                        s.audible = true;
                        const sound3d = game.Levels3DPreview.sounds[sound.id];
                        const distance = sound3d.mesh.position.distanceTo(game.Levels3DPreview.camera.position) * game.Levels3DPreview.factor;
                        let volume = sound.document.volume;
                        if ( sound.document.easing ) volume *= this._getEasingVolume(distance, r);
                        if ( !s.volume || (volume > s.volume) ) s.volume = volume;
                    } else {                        
                        for ( let l of listeners ) {
                            if ( !sound.source.active || !sound.source.shape?.contains(l.x, l.y) ) continue;
                            s.audible = true;
                            const distance = Math.hypot(l.x - sound.x, l.y - sound.y);
                            let volume = sound.document.volume;
                            if ( sound.document.easing ) volume *= this._getEasingVolume(distance, r);
                            if ( !s.volume || (volume > s.volume) ) s.volume = volume;
                          }
                  }

Basically i have a 100+ lines override to change the distance = ... line

[vague umbra]

Sure. Looks like I did actually have to make a change for v10. https://github.com/mtvjr/background-volume/blob/master/scripts/volume.mjs

Lines 22 - 24 would be the likely culprit:

for (const mesh of canvas.primary.videoMeshes ) {
    mesh.sourceElement.volume = newVolume;
 }

My hooks might get modified too, but I imagine they would stay the same:

// Have the updateBackgroundVolume function be called when the ambient volume changes
let orig = game.settings.settings.get("core.globalAmbientVolume").onChange;
game.settings.settings.get("core.globalAmbientVolume").onChange = (...args) => {
    Logger.log(Logger.Low, "Ambient volume changed.");
    let ret = orig.apply(this, args);
    updateBackgroundVolume();
    return ret;
}

[calm sonnet]

FYI, the next V11 release will fix something in the onChange handler (10130). This is what the function will look like. It will probably change again in V12.

      onChange: v => {
        if ( canvas.ready ) {
          canvas.sounds.refresh({fade: 0});
          for ( const mesh of canvas.primary.videoMeshes ) {
            if ( mesh.object instanceof Tile ) mesh.sourceElement.volume = mesh.object.volume;
            else mesh.sourceElement.volume = v;
          }
        }
        game.audio._onChangeGlobalVolume("globalAmbientVolume", v);
      }

[fiery kiln]

There is (well, will be) one issue with this code in V12. Since the Sound and the AudioContainer are now the same object instead of two different objects, the following would change:

        const {
-           container: { sourceNode },
+           sourceNode,
            context,
            id,
        } = sound;

Because you want to use sound.sourceNode directly rather than sound.container.sourceNode

[fiery kiln]

If you want to create a GitHub issue for this (so I don't forget) I can factor out something that will make it easier to adjust the effective distance. This code has already changed some in V12 actually:

  _syncPositions(listeners, options) {
    if ( !this.placeables.length || game.audio.locked ) return;
    const sounds = {};
    for ( let sound of this.placeables ) {
      const p = sound.document.path;
      if ( !p ) continue;

      // Track one audible object per unique sound path
      if ( !(p in sounds) ) sounds[p] = {path: p, audible: false, volume: 0, sound};
      const s = sounds[p];
      if ( !sound.isAudible ) continue; // The sound may not be currently audible

      // Identify the closest listener to the sound source
      let minD2 = Infinity;
      for ( let l of listeners ) {
        if ( !sound.source.active || !sound.source.shape?.contains(l.x, l.y) ) continue;
        s.audible = true;
        const d2 = Math.pow(l.x - sound.x, 2) + Math.pow(l.y - sound.y, 2);
        if ( d2 >= minD2 ) continue;
        minD2 = d2;
      }

      // Determine the resulting volume
      if ( minD2 === Infinity ) continue;
      s.distance = Math.sqrt(minD2);
      let {volume, easing} = sound.document;
      if ( easing ) volume *= this._getEasingVolume(s.distance, sound.radius);
      s.volume = volume;
    }

    // For each audible sound, sync at the target volume
    for ( let s of Object.values(sounds) ) {
      s.sound.sync(s.audible, s.volume, options);
    }
  }

The logic is improved to:

1. First iterate over potential listeners and find the closest one using squared-distance to avoid unnecessary Math.sqrt computation.
2. Then determine the effective volume based on the minimum distance

We could do something like factor the interior bit out as a helper function like _getEffectiveDistance(sound, listeners) or something.

[flint cape]

Ok that’s easy enough I’ll make a note

[flint cape]

So, I would need all the sounds to calculate distance, not the already “filtered by closest” list tho

And I would really love to remove this massive override for a simple distance method wrapper

[fiery kiln]

So, I would need all the sounds to calculate distance, not the already “filtered by closest” list tho
That doesn't seem to be the case from the code you posted, unless there's more to it.

[flint cape]

So, if the sound has the “3d sound” flag, I’ll do the 3d distance calc

But if your new quad tree filters out the sound beforehand it would be a problem

Because the sound can be very far from a token but still close to the camera

[calm sonnet]

My Limits module is also forced to make a massive override of SoundLayer#_syncPositions: https://github.com/dev7355608/limits/blob/main/scripts/patches/sound.mjs#L37-L43
In my use case I need to apply volume adjustments based on the sound origin and listener positions.
So perhaps we need something like _getVolumeMultiplier(soundSource, listener) and _getMaximumVolumeMultiplier(soundSource, listeners)

[flint cape]

I’m overriding the whole SoundLayer#refresh

[fiery kiln]

it's not a competition! 😛

[flint cape]

I don’t remember why at the moment, but it might be because I have to touch a private method or something

I’ll investigate once I’m at the airport and can get out a laptop

[fiery kiln]

Not urgent, we have plenty of time - but I do encourage you to open a GitHub issue and see if we can reduce the blast-radius of what you have to patch. Same for @calm sonnet use case with limits.

[flint cape]

actually i'm not sure why i'm overriding SoundsLayer#refresh, is it possible that at some point syncPositions was protected?

@calm sonnet if you want to add to this https://github.com/foundryvtt/foundryvtt/issues/10134

[cunning drum]

I feel like I am the target of all the breaking changes. 😅
This will severely impact SWIM because I make heavy use of AudioHelper.

[wooden loom]

In what way? I searched through your repo and you primarily only use AudioHelper#play, which is unaffected by the changes

[marsh sorrel]

Walls have Ears will definitivelly be impacted, I still want to retain compatibility so maybe a maxVersion is in order

[wooden loom]

Unrelated to the API changes, but are there any plans to add API functionality to play one-off sounds at coordinates, similar to ambient sounds?

[fiery kiln]

Nice idea! Could you write it up as a github issue with a brief summary? It's topically related to other V12 changes so I think I could do it.

I've also been meaning to convert door sounds to such a system where the sound of the door interaction is a localized effect instead of played for all clients, so the core door sounds could use such a method.

[fiery kiln]

If you can share a few examples of macros or code where you think you're impacted I'm happy to help you look into it and determine.

[fiery kiln]

Sounds good, please feel free to chat with me if you have any questions about how you might approach it in V12+, although I'll say that there are probably some more changes coming that are less API-related and more user facing that might end up also affecting what you'll want to do with the module in v12.

I think you are probably more impacted than any other module author by this change (sorry!), so please feel free to work with me to figure out the best solution and path forward.

[wooden loom]

https://github.com/foundryvtt/foundryvtt/issues/10137

I hope this helps!

[fiery kiln]

Great!

[marsh sorrel]

I will, thanks a lot

[marsh sorrel]

Do we have a maintained types library for Foundry API? I know there was one at some point

(yes this is related to the audio API. It could speed our trial & error process of updating our modules)

[fiery kiln]

No, BUT the renovated audio API improvements mentioned here are the first of a set of changes which shift client-side Foundry JS to ESModules. As part of that process the API concepts are:

1. Rigorously documented using typedoc
2. Exported from that ESModule

A result of this is substantially improved IDE comprehension of type hinting. You should also be able to build a valid .d.ts` file from the ESModule if you wish.

[marsh sorrel]

That works, thanks , many mod devs will be grateful of having an actual API to program against, kudos

[fiery kiln]

Some exciting related features that build upon this set of changes: https://github.com/foundryvtt/foundryvtt/issues/10152

@marsh sorrel this might impact the way you want to provide module functionality in V12, depending on whether you want to integrate with the new core "Special Effects" section of sound configuration.

[fiery kiln]

whoooosh https://github.com/foundryvtt/foundryvtt/issues/10153

[wooden loom]

Fantastic!

[marsh sorrel]

This is awesome news. Almost effectively audio muffling is integrated into core. Just a headsup:

• Sound's muffling level is dynamic in nature. Maybe a "muffling contribution" in wall parameters is better than (or in addition to) setting a fixed value
• Maybe the same muffling contribution when the sound is not in the same layer (level) as the token (which is your listener)
• Muffling fixed value is good for underwater environments
• Muffling frequency is exponential not linear, just be aware of that
• Recycle (changing values) filter nodes as much as possible, changing the node graphs almost always has a glitch sound much like the glitch sound of unbuffered looping

Also had an algorythm in mind:

• When evaluating the filters for a sound do this:
• Shoot (say) 24 rays in all directions until they collision in a wall
• Mind the muffling contribution of that wall (ethereal walls or terrain walls dont matter)
• It will generate <= 24 collision points (bear with me on this)
• From those points figure out if there is a straight unobstructed line to the listener (token)
• If there is an unobstructed line, apply a reverb filter (sound is bouncing on a L shaped corridor wall)
• If there is not unobstructed ray, then we apply muffling effect with the level calculation of muffling contributions of the walls in the ray cast

(you can estimate big hall reverberance if all 24 rays have hit a wall (enclosed environment) and the reverb level is marked on the average distance of all 24 rays)

(distance meassured in FT according to the unit transformation of the scene)

[fiery kiln]

Good feedback - I hope that you'll be able to engage with us some during the prototype phase once a release is available to help review, provide feedback, and test what we've done.

I hear and agree with your points. One thing I want to emphasize is that a design goal for me with how this appears in the core software is to present it in a way that users do not have to have knowledge or familiarity with audio concepts in order to use the system. I think the V12 core approach will be intentionally simplified, but a goal will be to make it easy for modules to do more complex or sophisticated things.

A key area of feedback I'd love to collect from you is whether you think the provided API makes it easy enough for you to accomplish a more advanced design that replaces the comparatively simpler core solution.

Recycle (changing values) filter nodes as much as possible, changing the node graphs almost always has a glitch sound much like the glitch sound of unbuffered looping

A question on this specifically in case you have advice. I've worked with both models. Certainly with BiquadFilterNode you can toggle it on or off by setting the type to "allpass" which allows you to disable the effect without rebuilding the audio pipeline.

For other effect types like ConvolverNode however, the node does not offer a parameter that can temporarily suppress it. You can manage it using a separate gain node though.

I haven't noticed obvious issues on my end when rebuilding the audio pipeline (when necessary) but my CPU is very fast so my PC is not the most representative test case I think.

[marsh sorrel]

Right, but is very noticeably in other setups, is not a matter of speed, I think the entire pipeline is sent to the sound board and that's why the milleage may vary. I solved it by having a cache of references to the nodes and changing the muffling level on the references nodes, it works like a charm, because even reattaching the Destination node will make it glitch.

Maybe you can set everyhitng up and bypass it with subclassing the ConvolverNode and adding a disable flag

[fiery kiln]

That's basically what I've already done, so I encourage you to check out V12 Prototype 1 once it's available (not for a few more weeks) and then let me know your thoughts. I would appreciate that.

[marsh sorrel]

I certainly will, looking forward to it!

[fiery kiln]

bump

[marsh sorrel]

@Here can you shed some light on how the filters are being assigned based on wall interference?

I'm pondering the usefulness of the add-on as it seems it got absorbed into the core system (which is fine for me but I want to check for setting compatibility/deprecation)

[fiery kiln]

There will still be lots of opportunities I think for a module here, if you want there to be.

The logic for when a sound is “muffled” is fairly naive: does the sound collide with at least one wall along a direct ray between the origin and the listener. There are more sophisticated models that could be used.

Also room for modules to add different filter types to expand the set of options users can choose from.

[marsh sorrel]

Awesome, I would like the sound filter assignation would be overridable at lease. I can then tamper into it and not consider it a hack.
I can then apply my logic:

• I have a muffling level formula (see image bellow) I can make that happen in the new system
• I can add (fake) sound bouncing on the walls (with reverb and echo/delay)
• Add support for multiple levels and sound muffling on ceiling/floors
• Detect balcony type scenarios where the walls restrict movement (and potentially vision) but not hearing
• Support unidirectional walls
  But basically I need to be able to disable or rewrite the raycast logic, the filter assignment/enablement logic and not feel like I'm hacking through it

WHE is intended to be as simple as possible.

[fiery kiln]

I’m not certain your requirements will be met with the code as it stands now, but I think they can be. I hope you’ll have a chance to look at Prototype 1 to get a general sense of how you could integrate custom handling into the framework

[fiery kiln]

“Intended to be as simple as possible” 😉

Just teasing, there are some cool algorithms here that can be applied

[marsh sorrel]

Well, yes, it has mufflin levels but those are opinionated and it doesnt have many (if any) controls on the config

[marsh sorrel]

This is my testbench

[limber walrus]

I use a lot of new Sound("/path/to/file.mp3").

What's the recommended approach atm?
the issue mentioned that foundry.audio.Sound is available or should I use the AudioHelper?

I know there was some unexpected behavior (or error) with audio helper is why I ended up using Sound but don't remember what that was.

[fiery kiln]

The Sound global class which you are using now is moved to foundry.audio.Sound - and improved along the way, so you will eventually need to migrate but what you're doing now should continue to work in V12.

The following are equivalent:

await game.audio.play("path/to/file.mp3");

const sound = new foundry.audio.Sound("path/to/file.mp3");
await sound.load();
sound.play();

[marsh sorrel]

After careful consideration I think there is still a place for WHE in the Foundry Ecosystem. The base platform now have taken the decision of needing the sounds to be NOT constrained by walls (I still need to figure out what it means for the WHE raycast). But in all this is my todo list:

- Move the project to parcel/TS
- Implement if possible the types on the new framework
- Update Yarn and automated workflows (autodeploy?)
- Add a global or scene setting to "handle muffling intensity by wall estimation"
- Prevent showing the Muffling intensity slider
- Possibly prevent showing the Muffling selector ( or auto assign it)
- Change the intensity slider just for a given user and not be a server setting
- Handle new types of walls and the proximity/reverse proximity cases
- Create entire new tutorials and test bed