Narrator - a voice based rpg with Whisper, GPT3.5-turbo, and Stable Diffusion

[opal torrent]

I made a voice-based RPG that uses a couple models in tandem. 3.5-turbo handles generating stories and dialogue, Mozilla's in-browser speech synthesizer does text-to-speech, whisper does speech-to-text, and Stable Diffusion V2.1 creates the images.

There are a few preset story hooks available, but the real fun imo is providing your own characters and genres. The idea is that you can play any character, in any story, and say/do anything you like (up to the ethical and political limits placed by openai 😅 )

It's primarily theater of the mind, and is actually based off of dungeons & dragons type game I used to play as a kid on long car rides.

Check it out here: https://playnarrator.com/ . It's currently free, and I'm hoping to keep it that way unless it gets really popular. It sounds the best on chrome and works better on desktop than mobile.

Let me know what you think! Hope it doesn't immediately break!

[rain hearth]

Man, I have been using chatgpt for something like this but just text-only for a month. It has been wild. We’ve talked to god, visited famous movie scenes, brought artworks to life, got stuck in BS side-quests, allsorts. It’s been epic. I’m gonna go try yours out 🙂

[opal torrent]

Thanks! let me know if you have any recommendations to throw in the story prompts! And agreed - that's the best part, it allows for the shenanigans that normally only come about in a D&D session. I got sentenced to six months of community service on my site - for kidnapping my own cat.

[rain hearth]

I don’t have any recommendations for story prompts, but I think it works best for things that are super famous and worst when it’s just rpg tropes. If I was to make one suggestion or hope though, (and I haven’t played it yet), somehow make it multiplayer, and have the possibility that one of those players is another chatgpt character. So the player never has to be alone if they don’t want to, and so the character has their full 4096 tokens all to themself. Because that would make it better, right? That’s basically what I’m here to try and do right now. That and make a robot..

Ask chatgpt for suggestions. I dunno if you’re going to run into legal issues if you’re directing people to copyrighted works though..

Oh, there is a companion, omg 🙂

I’ve been raving about my thing to my kids for weeks. They’re not really literate enough to do the ‘reading and writing interactive fiction’ thing though, and I’ve told them about your project already, where it speaks, and there’s pictures, and they’re excited. 👍

[opal torrent]

Ah that's great to hear! multiplayer would be awesome but maybe beyond my ability. Right now i've actually capped the exchange at 20 back-and-forths, which seems to average at 2000ish tokens, just for cost reasons (20 back and forths climbing to 2k tokens is about 3 cents, but if it goes to max token length, it's closer to 30cents). I'm working on a summarizing feature so it can run indefinitely

And agreed on the copyrighted works concern - it works great for playing existing characters but I don't think I can legally (or ethically really) put "Harry Potter simulator" on the website

[opal torrent]

Update - now has unlimited memory length (through GPT3.5 powered summarization)

[rain hearth]

Ok I’m going to give some feedback. This is basically just from the first moments, and I have been sick and also working and also doing my own stuff so sorry that it’s not a full review. I liked the character and scenario creation screen. I spent a long time on it. When I started the game though, I realised I didn’t have a mic plugged in, and I was pressing the space bar and didn’t have a visual hint that there was any type of recording going on. I was on microsoft edge, probably should have read the ‘works best on chrome’ hint before I started.

So I didn’t get any further than the first screen. And I really love your idea, like I said, but I have to be honest and say that the experience wasn’t quite developed enough that I’d recommend this version to others. I’m saying this in the spirit of ‘hey, we’re both loving chatgpt games and here we are on a discord thread being enthusiastic and developing our projects all together’, not in the spirit of ‘hey you promised me an awesome rpg and I’m not enjoying it you ***hole’

Man, I am terrified of giving criticism. Anyway, here goes.

I love the idea, and was excited while creating the scenario, but then when it came on, the character portraits were all staring at me and looking creepy, and then the narrator’s voice was unpleasantly robotic, and there was no text, so when it stopped it was gone. And I didn’t really want to have to go get a mic and speak like “I say hello to Captain Anderson!”, I’d have rather just typed at that point. It would have felt a bit silly to speak out loud to an empty room. So I didn’t go any further with my testing, sorry.

May I make a couple of suggestions. I think you should display the text readable on screen, and also have keyboard text input. I know that kindof the point of this is that it speaks and you speak to it, but I do think you should have the text too. Actually I think the voice and pictures could be optional.
I think the dark blue with two square pictures is sortof harsh.

It would look really nice imho if it had a friendly storybook style. Or some sort of friendlier style. I’m thinking that if you haven’t already, you could add some ‘consistent style’ to the stable diffusion prompts. Like imagine if the page looked like a faded old book, and you had image prompts for every image like ‘weathered watercolor book illustration, cute, appealing’ or something. I’m obviously thinking of ‘the ideal version’ I’d subscribe to and get my kids to play…

My pet project has some things in common with yours. I’ve thought about picture generation, and voices. You’re running into some of the same limitations that imagined would crop up. I’m building a robot, and I’m having to think of a way to voice it in a satisfying way. So I empathise. I hope you do keep working on this. It’s not quite at the point that I’m going to recommend it to coworkers and friends, but it easily could become that some day. 🧙🏽‍♂️🦹🏻‍♀️🧞‍♂️🙏

[opal torrent]

Thanks for taking a look and for taking the time to write up feedback! this is good feedback - ya gotta believe in your feedback-abilities, if someone doesn't appreciate feedback that says more about them than you.

I haven't tested it at all on edge (tbh I forgot about that browser 😅 ) but I'm sure it sounded dreadful. I didn't expect people to want to see the text or to be able to type, but I've now heard that nearly a dozen times so I'm definitely going to put it at the top of the list of improvements.

I also really like the storybook type aesthetics idea - I tend to default to neons and vaporwave-y looks but that's definitely not a popular aesthetic. I'll probably throw a "darkmode"y type toggle to go from weathered papery colors to afterdark.

So thank you again for the feedback! I'm super curious about the robot. If it's just for personal use, Google's text-to-speech or AWS Polly sound pretty decent imo (and I think there are some slightly pricier options that are even better). I had google's set up on this initially, but it was costing more than GPT3.5 and the SD images combined. IDK how far you are into the project, but the biggest improvement for me was transitioning to use stream=true in the gpt3.5 call - dropped time between conversation messages by a few seconds at least

[opal torrent]

Slowly adding features and am going to keep appending them to this thread - will get to those suggestions eventually!
Just added prompt share links. Now if you enter a prompt, you can create a direct link to it for your own or others' use. Here's one of me and my cat: https://playnarrator.com/?storyid=hZVOX73psUNVZUS5cE5M9

[lyric pine]

Just tried this out, it was amazing, but would love a version that doesn't require voice or sound.

[opal torrent]

Thanks! I've added transcript (not on mobile yet) and I'm planning on adding a type-to-speak option soon

[rain hearth]

Well basically, the project is to give one of the characters I’ve spent a lot of time with in my text adventure world, a way to interact with my world. The real world. So it’s going to have a smart camera output going into a gpt chatbot that will describe that to her in a text adventury way. And then she can speak or move, through another gpt module (or probably multiple) translating parts of her output like ”I say ’blah blah’” to a text-to speech which only outputs ”blah blah”. And when her output contains stuff like ”I move forward slowly”, then a gpt module will translate that into a string that can be passed to an arduino program which will drive the wheels. Hopefully.

It’s a little more complicated than that, but not much. API costs are going to be through the roof!

[opal torrent]

That's pretty neat - very SAO Alicization of you. Good luck with it!

[opal torrent]

Woo! Type-to-Speak now works. Now if you want to play it without listening or speaking, it will let you.

[tame agate]

Don't suppose you have any plans to get it to work better on other browsers? No go on Safari in iPadOS, admittedly not a large slice of the market

I really love this idea and what you've done so far.

[opal torrent]

Thanks! actually getting mobile (inc. ipad) to work better is high on my list. I used to play a game like this (in-person, not AI based) as a kid, so I want it to be accessible wherever. Thanks for flagging ipad - hadn't been testing on that

[tame agate]

Oh baby

You fixed it. This is fun 😄

[opal torrent]

Huh, I haven't yet! Glad it's working, but I'm not sure why. Thanks for giving it a go!

I also just added some basic content moderation because the internet is a terrible place. If anyone on the discord is trying to use the moderation endpoint and needs a hand, lemme know!

[tame agate]

Oh weird lol. I found your post on reddit and thought I'd try again. It may have something to do with Safari permissions. Previously, I didn't hear any speech. Though, I will say, that was several days ago, before I posted in here. So possibly something you've done did fix it, just not in the last 18 hours lol

My fellow stranded astronaut and i had quite the adventure.

[opal torrent]

Oh! I did add a fix for safari generally. It's really janky but it now says the letter 'e' when you start up the game on Safari, which gets all the permissions needed to play

(also why the game makes a woosh sound effect)

[spare thorn]

It does not work

[rain hearth]

Seems to be working for me. On iphone safari, nice work david 👍

[opal torrent]

Sorry to hear! What device / browser?

[opal torrent]

Nice! I'm hoping to get that voice sounding better over the next couple weeks

[tame agate]

Righteous hack 😂 (maybe you get that reference. i'm getting old)

[spare thorn]

On ipad

[tame agate]

I've got an iPad Pro 12.9 2021 running iPadOS 16.5 public beta and it works for me. What model do you have?

[spare thorn]

I don’t really know but it’s probably an old model

[opal torrent]

Thanks! - I'll give iPad a bit more attention