Issue with running dynamic python code as async using pythonnet

[dapper meteor]

public static async Task<string> GetMicrophoneTextFromWhisper(string path)
        {
            if (!File.Exists(path))
            {
                Debug.LogError("Audio file not found: " + path);
                return null;
            }
            string transcription = string.Empty;
            using (Py.GIL())
            {
                dynamic whisper = Py.Import("whisper");
                dynamic model = whisper.load_model("base");
                dynamic transcriptionResult = model.transcribe(path); //awaiting this never finishes
                transcription = transcriptionResult["text"];
            }
            OnTranscribed?.Invoke(SpeechResult.New(transcription, filePath));
            return transcription;
        }

Hey all, I am having an issue with this speech to text program. It runs fine, I am just trying to optimize it. Since the model takes a second to process the audio file and I don't want to stall the main thread for that long I tried awaiting it but it seems to never reach the line under it. Is there some other way I can optimize this without Task.Run? From what I know I can't find a solution. Only thing I can find is making the python function async but I tried that too and the same result is happening so I am not sure what the best way to make this fast would be. Any thoughts or solutions would be helpful, thanks 🙏

Profiler confirms it

^this is without awaiting the transcribe call

[gentle hinge]

Normal async await in unity is single threaded. You need to use Task.Run or run your code on a separate thread some different way.

[dapper meteor]

im aware thats kinda the issue tho. when i use Task.Run it never finishes. It must be conflicting or not supported or whatever it may be. But I can't execute the whole thing off the main thread either because then I just get messages saying you can only do xyz on the main thread. Then back to square one

[gentle hinge]

Then don't do things that require the main thread in that code. This should only be unity API, so you should be fine as long as you just run inference on the model.

[dapper meteor]

inference? And in the code you see I don't. If I execute the transcription with Task.Run it never completes

[gentle hinge]

You should investigate the errors that you're getting and handle them appropriately instead of looking for an illusive solution that doesn't exist.

[dapper meteor]

there is no errors.

[gentle hinge]

Wdym by "it never completes" and how do you know that"?

[gentle hinge]

You mentioned errors about not running on the main thread.

[dapper meteor]

I put debug logs after each step and when it is not wrapped in a Task.Run it runs fine (with the stutter ofc and the logs print) and when I do use Task.Run they stop printing once the task gets run and never completes

[dapper meteor]

I was talking about when I ran the entire thing off the thread not just the transcription

[gentle hinge]

I'm not sure debug logs run outside of the main thread. You'd need to debug with a debugger.

[dapper meteor]

well its still the main thread

let me show the difference

[gentle hinge]

Task.Run runs the task on a background thread

[dapper meteor]

        using (Py.GIL())
        {
            // MAKE SURE FFMPEG IS INSTALLED `choco install ffmpeg` its too big for git
            dynamic whisper = Py.Import("whisper");
            dynamic model = whisper.load_model("base");
            dynamic transcriptionResult = model.transcribe(path);
            transcription = transcriptionResult["text"];
        }

        using (Py.GIL())
        {
            // MAKE SURE FFMPEG IS INSTALLED `choco install ffmpeg` its too big for git
            dynamic whisper = Py.Import("whisper");
            dynamic model = whisper.load_model("base");
            dynamic transcriptionResult = await Task.Run(() => model.transcribe(path));
            transcription = transcriptionResult["text"];
        }

pretend theres logs between everything

[gentle hinge]

The second snippet is just gonna stall the main thread.

You should not await it here. Get the task and then await on it async on the main thread.

[dapper meteor]

        using (Py.GIL())
        {
            // MAKE SURE FFMPEG IS INSTALLED `choco install ffmpeg` its too big for git
            dynamic whisper = Py.Import("whisper");
            dynamic model = whisper.load_model("base");
            var theTask = Task.Run(() => model.transcribe(path));

            while (!theTask.IsCompleted)
                await Awaitable.EndOfFrameAsync();

            transcription = theTask.Result["text"];
        }

something like this?

I see what you mean though

[gentle hinge]

Yes

[dapper meteor]

yeah idk

i tried

        using (Py.GIL())
        {
            // MAKE SURE FFMPEG IS INSTALLED `choco install ffmpeg` its too big for git
            dynamic whisper = Py.Import("whisper");
            dynamic model = whisper.load_model("base");
            Debug.Log("starting task");
            Task<dynamic> theTask = Task.Run(() => model.transcribe(path));
            
            while (!theTask.IsCompleted)
                await Awaitable.EndOfFrameAsync();

            Debug.Log("task complete");

            transcription = theTask.Result["text"];
        }

starting task prints I waited like 2-3 minutes it never completed
usually takes like 4 seconds max

[gentle hinge]

What is model.transcribe? What api is that?

[dapper meteor]

its whisper openai

https://openai.com/index/whisper/

[gentle hinge]

I don't think they provide C# api..?

[dapper meteor]

yeah they dont lol

[gentle hinge]

So it's not

[dapper meteor]

wdym its not

thats why i have to do this crappy pythonnet solution

[gentle hinge]

It's not whisper. At least not directly

[dapper meteor]

well I import it and use its functions

what else could it be

[gentle hinge]

Import how? What api is that?

[dapper meteor]

thats pythonnet

using Python.Runtime;

[gentle hinge]

Ok, so nothing to do with unity. Meaning it should run properly on a non main thread

[dapper meteor]

so im doing something wrong then

[gentle hinge]

So you run that model locally?

[dapper meteor]

meaning?

[gentle hinge]

You run inference on your GPU? Or are you making a request to open ai api?

[dapper meteor]

Im not sure its whatever the transcribe method does im trying to find if it requests right now

but based on previously knowledge it shouldnt be a request

[gentle hinge]

These are some very basic questions that you should have know an answer for before even starting writing the code...

[dapper meteor]

i mean that one is not very self explanatory 😭

[gentle hinge]

I'm looking up a bit now, and it seems there are many integrations of whisper both with C# and even specifically unity. I have no clue why you choose to write your own custom solution

[dapper meteor]

where do you see this

[gentle hinge]

google

[dapper meteor]

i looked i found nothing

[gentle hinge]

well, perplexity to be specific

Here's one:
https://github.com/srcnalt/OpenAI-Unity

Ah but that's for using their web api

[dapper meteor]

and thats also not speech recognition

doesnt look like whisper is included

[gentle hinge]

For ai model inference on local machine, there's unity sentis. An official package by unity.

Unity Sentis Integration
Unity Sentis, a neural network inference library, provides a way to run Whisper locally in Unity projects:
Unity has optimized the Whisper tiny model for use with Sentis.
The model can be downloaded from Unity's Hugging Face page.
This integration allows for offline speech recognition directly in Unity applications.

Here's the link to the model itself:
https://huggingface.co/unity/sentis-whisper-tiny

The instructions on the page actually explain it all

[dapper meteor]

wow

what a find

im gonna try to set this up I'll report back

thx for finding that im pissed I didn't

[dapper meteor]

Ok so I have it set up, but for some reason it just prints gibberish when I run it. I did exactly what the directions said. Theres no errors, its just not printing right or something. If the models aren't working im not sure what I can do because I don't have experience with sentis I am not sure what I would need to modify to get this working. Here's the script from that website:

https://paste.ofcode.org/3b5qp54jVdDd4wsQ6PQBhYt

[dapper meteor]

Nevermind think I figured it out

[gentle hinge]

What was the issue?