PreProcessing AudioData

[hexed nacelle]

does anyone know any way to pre process useAudioData function on python or JS to make the AudioData value and pass only the down sampled value in config??

[torn nebula]

the audiodata type is just

export type AudioData = {
    channelWaveforms: Float32Array[];
    sampleRate: number;
    durationInSeconds: number;
    numberOfChannels: number;
    resultId: string;
    isRemote: boolean;
};

so you can externally modify the array, how to modify a waveform i don't have much experience though

[hexed nacelle]

I don't want to modify it
I just want to pre process before render
and pass the data in the config
but I still couldn't find a way to do it in Node js or any package

[torn nebula]

what is preprocessing? reading it? downsampling is modifying it

I am looking into whether we can make an API for resampling!

[hexed nacelle]

We want to be able to process the audio data only once before the actual remotion render
and pass the data of get-audio-data in my config to
so in the our code we build an AudioData instance and use it like we use it now
there are two reason we want to do it

1. we want to remove the multiple get-audio-data executaiton ( once per each lambda function )
2. in our product we have for example an hour video that we use the media to render 20 second of it with specific start_second and end_second

• the audio data will run on the whole get-audio-data which is not nessessery

[torn nebula]

@hexed nacelle have you considered using useWindowedAudioData()?

I understand your request, this would be useful to have large data preprocessing, it's not implemented at the moment

[hexed nacelle]

Yeah I checked useWindowedAudioData but it only works on wav
but all our video are aac

is there any work around
for example running getAudioData once and sending an http request from the remotion render?

[torn nebula]

it is just a buffer, so you may make a design that this is cached over the network

we don't intend to solve this problem this way because we think we can make a function to make useWindowedAudioData() use webcodecs to support any audio.

right now I am working on the seeking part, just merged another container and now we support 6/9 containers, so I hope this is not too far away, and of course this would be a lot easier

[hexed nacelle]

oh that would be amazing

[hexed nacelle]

v4.0.289
does this version has useWindowedAudioData supporting other audio codecs?

base on the release not I guessed it's now support other codecs
if I am right
throw new Error('getPartialAudioData() requires a WAVE file, but the first bytes are not RIFF. ');
this is still happening

[torn nebula]

@hexed nacelle this release was just to fix this bug https://www.remotion.dev/docs/lambda/troubleshooting/chrome-133-situation

it was never meant to add support for other audio types, this function just supports wave as the documentation states

however, I just now finally finished remotion media parser, and if we combine it with the seeking features and webcodecs I think we can now extract a partial wave

https://www.remotion.dev/docs/media-parser/webcodecs#using-remotionmedia-parser
https://www.remotion.dev/docs/media-parser/seeking

we are ready now to explore these use cases and will do so, if you urgently want to implement this, you can also experiment with it

[hexed nacelle]

with Wav it only will download part of the audio right?