Mood/tone determination from speech-to-text

[visual ibex]

Something that just occurred to me is that Neuro & Evil are probably responding to pure text when talking with Vedal and collab partners, which sometimes may confuse them if the tone of the speech is not what the speaker intended, for example sarcasm or anger.

I wonder if whatever speech-to-text technology Vedal is using could be upgraded to detect the "tone" or the "emotion" of what's being said and fed as additional context to the twins to make them better at understanding how they are being spoken to, if that's even a technology that exists. It could also potentially help with translation to other languages if it's something that's also relevant to convey.

Unless it's already a thing and I just haven't noticed it, in which case I apologize.

[verbal kettle]

i'm pretty sure neuro and evil understand sarcasm

[visual ibex]

I don't doubt that LLMs are capable of that to some extent but there's only so much you can detect from text alone.

[limpid hawk]

i think a recent call-in with koko showcased they have some understanding of tone since neuro was able to detect sarcasm, at least. I dunno how much understanding they have or what extent they have for it, but it's at least there

i think it was something shown off specifically in a dev stream last year too

[ripe wadi]

I agree, I also I think they might Nuero might have gotten it from context and not audio detection. I hope she just gets the best tech tutle can manage.

[graceful atlas]

I actually thought Neuro-Sama and Evil actually detect speech directly as if hearing it.

So what's actually happening is that everything said to the AI sisters in voice is transformed into text that is invisible to us but read by the AI?