Twitch Subtitles Extention

[high fulcrum]

I’m curious how hard it would be to make a Twitch extension that provides subtitles in different languages, or whether it’s possible to use Twitch’s integrated Closed Captioning for this.

It seems like it wouldn’t be too hard to implement and would have a lot of benefits. You wouldn’t need to stream on different channels for different languages, everything could be available on the main channel, with the option to add even more languages later. It would also be possible to disable subtitles for those who don’t need them and make it easier to clip funny moments. However, in that case, VODs and clips would be without subtitles entirely, which could be a drawback.

This idea seems pretty obvious to me, so I’m curious what the downsides are, because I don’t really see any. The only issue I can think of is that it would probably work poorly in the Twitch app, where extensions aren’t shown directly on the player but are instead placed in the chat menu.

[upbeat edge]

I had a look at Twitch's closed captions docs and they aren't super detailed tbh, and I don't remember if OBS offers something for that as a solution. The extension route runs into adaptive design issues, something clearly Vedal is good at

Both options also suffer from potential audio/caption desync, integrated closed captions likely not so much since you're probably getting the captions from the same laggy but distributed servers, but it can get worse for both viewers and Vedal if the extension route is taken, since now Vedal has to provide a server that will have to serve ~10k socket connections avg probably, from all around the world, so I don't imagine it having a fun time (probably, obviously don't know Vedal's server setup).

You also need to think about the slow backup filter, since it can and does cut off Neuro mid sentence if she's about to say something wild. By the time it finds something kinda, uh, bad, the subtitles may already be sent and there's no way to retract that, so you'll end up sending those subtitles anyways. Issue can kinda be solved if the extension route is taken but it would still be sent and depending on network connections it may stay on screen for longer than expected.

The issue with VoDs not having subtitles can probably be rectified by uploading the subtitles separately after stream (or probably in the VoD itself if using the Twitch CC stuff, I'm not too familiar with it as you can tell but from what I can see it might be possible).

An issue I just realised with multiple languages on one channel, besides the fact that now Vedal would have to compute and send everything separately, is that the different languages are now coerced into one channel, where most of those people are spamming something in English and it becomes hard to actually do anything in there.

holy yap how did I not get essaying reacted by neurobot

or evilbot or whatever rn

[high fulcrum]

I saw this example https://stream-cc.gooseman.codes/showcase and it works fine. I thought that with Vedal's existing TTS and translation systems, something similar could work

[grave granite]

i believe vedal spoke about this in the past, he said something about how it would allow people to collect neuros speech and use it to train their own “neuro” which he doesn’t want

[high fulcrum]

Can't people already record Neuro's speech if they want to? I don't see how subtitles would change anything

[halcyon radish]

There are even transcripts of every stream that ladev uses to search streams so doubt that's the reason

[grave granite]

idk why people are clueless reacting im just repeating what vedal has said
https://youtu.be/gnbWxNFi6rE?t=3907

[high fulcrum]

It's not that I don't believe you, it's just that the reasoning sounds a bit weird. btw thanks for the clip, it’s exactly what I was looking for

Wait, if he’s worried about the captions being used to train a fake Neuro, why not send them as images instead? I know the Hearthstone extension can load card images in real time, so maybe something similar could work with captions too

[split nest]

seems to me that he either changed his mind now or just doesnt care anymore because too many people did it
he even used ladev libary a few times and linked it so he def knows of its existance too

[split nest]

this was a thing even earlier than this at the very start early 2023 where people wanted a way to log neuros sentences for quotes and an easy way too look up funny moments
got shot down everytime someone tried

[ocean needle]

We don’t want to mix languages in the twitch chat. Plus you can’t integrate in the Billi Billi stream since I’m pretty sure twitch isn’t allowed in China.