OpenAI API Plugin for the VoiceAttack Program

[ocean nexus]

The OpenAI VoiceAttack Plugin provides a powerful interface between VoiceAttack and the OpenAI API, allowing us to seamlessly incorporate state-of-the-art artificial intelligence capabilities into our VoiceAttack profiles and commands.

I'm so excited to bring the power of true artificial intelligence to VoiceAttack through this plugin for all profile and command builders out there interested in working with OpenAI Technologies in VoiceAttack! We can use raw text input, dictation text, or captured audio from VoiceAttack as input prompts for ChatGPT, and we can receive responses as a text variable to use as we wish, or set it to be spoken directly and specifically tailored for text-to-speech in VoiceAttack. We can also perform completion tasks on provided input with options for selecting the GPT model (and more), processing audio via transcription or translation into (English) text using OpenAI Whisper, and generate or work with images using OpenAI Dall-E.

This plugin also features OpenAI Moderation to review provided input and return a list of any flagged categories. Lastly, we can use the plugin to upload, list, or delete files for fine-tuning the OpenAI GPT models, or make use of OpenAI Embedding, which returns a string of metadata that can be parsed and used as desired. With this plugin, we can access a wide range of OpenAI functionality with ease, directly from within VoiceAttack.

Find complete details, download link, discussion pages, and comprehensive Wiki documentation on the open source repo at GitHub:
https://github.com/SemlerPDX/OpenAI-VoiceAttack-Plugin
... also, I created the first VoiceAttack profile using it for vocal ChatGPT conversations we can create with speech-to-text and text-to-speech commands in VoiceAttack.

Hope you VoiceAttack users find this as entertaining as I do, especially hearing it in our own custom TTS voices, like my Ivona Brian (UK)
🍾 😁
https://veterans-gaming.com/files/file/43-avcs-voice-chat-artificial-intelligence-profile/

[ocean nexus]

BIG things on the horizon! Now that the plugin has been released, I've had time to circle back to the most advanced feature of the OpenAI Plugin for VoiceAttack: Embeddings

What are embeddings?
Embeddings are a way to represent a body of text as an array of numeric values which capture the meaning and context of the text, allowing for comparisons between different texts. The OpenAI API offers a very fast and very affordable means to get these numeric values for a block of text, which are called 'embedding vectors'. The total number of embedding vectors generated for text is always the same, and the OpenAI Embeddings provides 1,536 float vectors for any text content. With such a high dimensionality, comparisons can have an increased degree of accuracy.

How do we use embeddings?
Consider a database that contains many entries, each entry has various data fields, most importantly the text content and the 1,536 embedding vectors generated for that content. These are "float" number types, similar to a decimal. If the user asks a question, a system can first get the new embedding vectors for that question, and then compare those to all entries on file through something called "cosine similarity". Through that, we can discover each block of text in the database which is most similar to the question which was asked. Then, we can take any number of those most similar text entries and present THAT to ChatGPT (along with the original question), instead of only the question, and tell it to use the data along with the question to produce an appropriate response. By doing this, we can "feed" information to ChatGPT for which to base its response upon, for situations where it would NOT know this information (such as a help document, a wiki page, a book of short stories, etc.). Rather than using its own knowledge base, it can formulate an organic response using the provided data.

A personalized, local "brain" for our AI chat bots!

What is this? TLDNR;

• Introduction of local database processing for new user inputs before sending them to ChatGPT.
• Ability to ignore irrelevant questions and respond using existing knowledge base.
• Database selection, topic specification, and subject refinement options for ChatGPT context plugin calls.
• Support for adding, viewing, editing, and removing documents and individual entries in the database, including PDF format.
• Command Action system to execute commands directly in VoiceAttack when a user input matches a specific entry.
• Contiguous subject system for reading entries using text-to-speech, with the ability to pause and resume reading from where it left off.

How will this work in OpenAI Plugin for VoiceAttack?
A few new plugin contexts will be added to expand the currently lacking Embeddings context, as well as a handful of additional option VoiceAttack variables for the ChatGPT context(s). These will allow us to specific that new inputs should be processed against a local database, which would occur just after getting new user input and before sending that input to ChatGPT. When the question is NOT relevant to the similar text content provided to ChatGPT to help formulate a response, it will ignore it and just answer as it would normally using its existing knowledge base, else it will use the data to respond to the user input.

...
Before beginning a ChatGPT context plugin call, we can indicate the database to use, optionally a particular topic contained in that database referring only to a certain set of entries, and also optionally a particular subject of that topic to further refine the specifics of a particular call. By default, when not specified, the entire database would be queried.

Users will be able to add new documents in whole (even in .PDF format), or add individual entries. There will be a system to view, edit, or remove entries as well - individually, all entries, or by topic name, or topic + subject name.

An additional system will allow setting a Command Action value for an entry, and a way to indicate that when a user input matches an entry with such an action set, to execute the command directly in VoiceAttack rather than provide the user input to ChatGPT as a question to be responded to.

Another interesting new system will have a VoiceAttack variable we can set, indicating that Embeddings should treat all entries in a subject as contiguous. Once identified through contextual similarity to the user input, they can be read using text-to-speech entry by entry until paused/stopped, where it will save the index of where it left off in that topic + subject. This could allow us to feed a document to the database such as a book of short stories, where we could ask it to read one of them to us, or continue reading from where it last left off - all without contacting OpenAI API beyond the initial embeddings for the user input to match it to and identify the contents of the database to be read.

...
When will this be available?
Because this system will be introducing an SQL database layer to the codebase of the OpenAI Plugin for VoiceAttack, it will be awhile before I can feel secure adding this to the public branch of this repo on GitHub. I expect it may be late June or into July before everything is ready for prime time, so I intend to introduce an early Beta branch to the GitHub repository. This will allow interested users to begin testing and trying out this new system, and help out by give me the feedback I need to ensure performance and functions are consistent across all systems. If all goes well, I should have this Beta branch available in a few weeks, but again due to the complexity of this refactor, public testing and some feedback will be required before I merge it with the Main branch and push this update to everyone.

Pics or it Didn't Happen
So far, I have been testing the loading phase of the database which occurs once when VoiceAttack is loaded, and the cosine calculations of the new embedding vectors against a test set of 25,000 entries, with a goal of optimizing the speed of these functions. For reference, a database of such size would contain about ten 300+ page documents. I have gotten the loading of the database to just over 8 seconds for such a massive test database (down from 24 seconds!), and the calculations return down to just 0.189, all achieved by parallelizing these tasks across all CPU* cores on the PC:
https://i.imgur.com/lSosD2d.png
(I should note that my CPU is an AMD R9 3900X with 12 Cores and 24 Threads which this 'Malcom' VM above has full access to, so I will be keen to discover how optimized this will be on systems with fewer cores/threads)

Thanks for all the feedback and support so far - hope you all are enjoying the concept of real AI tools in our VoiceAttack profiles & commands as much as I am!!

[ocean nexus]

(glad this didn't get removed when I was removed for not being verified! nearly 200 users now, and no complaints yet!)

[kindred condor]

Awesome project! been hoping someone made this.

[shy granite]

Damn this plugin is more feature rich then some entire applications

[ocean nexus]

WOW! Over 500 users now of my OpenAI Plugin for VoiceAttack!! That's a lot of people who (also) enjoy true artificial intelligence systems in VA! 🥳 🍾
Check it out here:
https://forum.voiceattack.com/smf/index.php?topic=4519.0

[kindred condor]

only 500?

[ocean nexus]

Finally getting around to refactoring my OpenAI API plugin for VoiceAttack and it's been a hot minute since I've even looked at my code, release version is what people are using but I started refactoring and hadn't pushed anything yet, so it's in a quasi state...

Adding my new SQL database embedding vectors search function for documents and manuals - TL;DNR will be able to feed it any document(s) and then ask in natural conversational English questions and get natural responses based on any relevant data found in the database within a second or few ... speech input and text-to-speech responses, with an optional 'sources' list to review those areas of said document(s).

Testing with around 25,000 cosine similarity calculations, perfecting my 'turbo' mode that chunks it up into separate threaded tasks based on the available total thread count of the CPU. Super fast on my R9 3900X with 12c/24t -- and that's why GPU's are a dirty speed increase for AI style tasks, and why we'll soon see more affordable consumer grade PCIe cards for local AI tasks imho, at least I hope...

189ms is fast enough to not be the bottleneck in speed for such an interactive system that has other things it must wait for, such as offloaded LLM processing through the OpenAI API -- This would be the equivalent of searching through like 5 game manuals of around 500 pages each to find all relevant paragraphs to a natural English question pertaining to info in them in less than a quarter of a second.

[kindred condor]

I haven't looked at your codebase.. are you by chance using function calling, or standard chat completion with your own hooks?

the reason I ask, if its chat completion, nitro speed models (200+ t/s) exist as well

[ocean nexus]

I'm trying to make each OpenAI API function available from within the VoiceAttack program using a plugin, with some added functionality for flow control such as leveraging Whisper for speech-to-text user input, etc.

Everything (that uses OpenAI for a return) is a function call

[kindred condor]

okay, makes sense 🙂

[ocean nexus]

😲 🍾 🥳