Using Whisper and ChatGPT for my UI automation tool

[warm oyster]

https://youtu.be/t1PYks0UTL8
I'm loving the image generation for creating icons for my tool!
Also running whisperx locally for voice2text

[agile rover]

this is mind boggling, i am still stuck on what the purpose of this is

very interesting project

[warm oyster]

Working towards a simulator for LLMs to use UIs better

Currently working towards scraping vscode using ui automation in a way that can be replicated for other apps. Using a calculator step by step by pressing buttons instead of evaluating Python as a way for an llm to do provable computation or something

[agile rover]

so you want to make it so that the model

can actually interact with the computer?

[warm oyster]

yup

but in a way that also lets you sandbox it

[agile rover]

im just thinking that surely there must of been a easier way to do it haha

but this is super cool lol, just the infinite tabs throw me off

since its looking at a tab of game in the game

[warm oyster]

yea it's easy to get lost in the recursion

[agile rover]

is there a way to detect tge game inside the game and hide it? i assume not

[warm oyster]

possibly

zoomit.exe does it I think, so it should be possible

I also know the position of the window and its dimensions so I could censor it in the screen texture

but where's the fun in that

[agile rover]

I havent watched the full video, but im assuming you would take a screen shot of the screen the nsend it to the model for a response

then it would convert it into the controller inputs right?