#Game State API - Tooltip Twitch Extension

1 messages · Page 1 of 1 (latest)

mellow lintel
#

Hello!
I'm currently developing a twitch extension that allows viewers to mouse over cards while watching the stream.
This requires knowledge of what cards are on screen at any given time.

Suggestion

I think the best solution is to have the current game state saved to a file somewhere, this can then be read by another app.

There are a few other ways of doing this but each come with their own issues:

  • Mod: requires a lot of setup, disables steam achievements
  • Image recognition: issues with screenshot capture when in fullscreen + borderless, very inefficient, inaccurate especially with foil/holographic/polychrome
final acorn
#

Oh yeah, Dead by Daylight streamers use something like that so viewers can read what their perks do. It might be a little harder due to the cards wiggling around a little bit and coming in different editions so some sort of image recognition might have a trouble with it

mellow lintel
#

Game State API - Tooltip Twitch Extension

leaden ibex
#

Hi !
There some tips for you to gain a little time at first. Don't focus on reading the cards in hand, it's the streamer job to explain the cards effect too if they have seal or steel and these are simpler effect that you don't need to mouse over. You should just focus on reading the joker list this will be enough.

There is a mod in slay the spire where you can mouse over the relicts. But again it's a mod so it disable achievements. But the function itself may be worth seeing to adapt/search.

Same with the pack cards and shops, the streamer will mouse over himself in the shop or the viewer can ask this create interaction. Shops being a temporary I don't know if it's very useful to focus on these.
Good luck on your extension.

mellow lintel
scenic drum
#

Hey there, always good to see folks working on accessibility for new players! If you haven't already, I would recommend taking a look at the architecture for the Slay-the-Spire equivalent mod https://github.com/MaT1g3R/slay-the-relics. It, and every other similar twitch extension I've seen or worked on, uses a mod to read game state, and does require some setup from the streamer, but as you noted it's going to be orders of magnitude easier than doing image recognition.

Also, not to scare you off the idea, but the Twitch review process for getting extensions approved usually entails quite a bit of back-and-forth with their staff, and for whatever reason they seem to have stopped handling new requests recently. Could be staffing cuts, reduced focus on the developer program, any number of things, but I would be prepared for the possibility that you may need to put the project on hold for a while. If you do want to get it in front of an audience before Twitch figures their process out, one thing I've seen an extension developer do is create their own embedded Twitch player website with the overlay built in.

vague rune
#

yo @mellow lintel, have you started work on this? i plan on prototyping some stuff today

Getting data into the extension seems fairly trivial, whatever runs on the broadcaster's computer can send stuff to Twitch's PubSub service which propagates to all clients: https://dev.twitch.tv/docs/extensions/building/#broadcasting-via-pubsub (note: max 1Hz, 5KB).
Ignoring authn for now, guessing a tiny backend service needs to exist for the purpose of giving long lived tokens to broadcasters.

The run save file (save.jkr) seems to have all the "card areas" (deck, discard, hand, jokers, consumables) and tags, it gets updated at 0.1Hz, it's deflate-compressed lua in the format return { ... }, so a standalone program just reading that file and sending it to PubSub looks fairly good.

FWIW, a mod would also work, there's an http library loaded which should be all that's needed. (W.r.t achievements, I don't know about the plans for the Steam workshop, but you can re-enable them in Steamodded by installing AchievementsEnabler.lua, the decision is the player's to be made)

mellow lintel
vague rune
#

I'm keen to work on the application if you'd like :)

I'd be weary of using the config service for those data transfers, I can't see usage limits and I'm guessing this isn't really the suggested use.