GPT and vectorstores

[plain carbon]

I'm planning to use a vectorstore to store embeddings to provide context between API calls. What I'll do is extract embeddings from GPT's response and update the vectorstore with those embeddings. Then after a new prompt, I'll grab vectors from the vectorstore that match the prompt and append those vectors to my new prompt so that GPT will have background to understand the history behind the prompt. My question is: will GPT understand what it means if I include vectors in the prompt as vectors?

for example:

[0.821,-0.126,0.553,...] dragon
[-0.345,0.198,0.721,...] talons
[1.038,0.288,0.099,...] sword
[0.845,-0.273,0.621,...] Sir Gallant
Describe how Sir Gallant fought the dragon and won.

Will GPT understand how the vectors relate to the prompt and respond appropriately?

[heavy bluff]

no and one embedding vector is a massive amount in plain text

[glass bear]

These vectors are used to compare the similarity of different words or sentences.

[south bloom]

Hi @plain carbon

I had the same confusion when I was starting out with LLM development 😅

To answer your question, no you wouldn't put the vectors themselves in the prompt. You'd rather put the actual texts corresponding to those vectors in the prompt, the vectors are just used to find similarity.

A typical flow goes like the following

1. There's a prompt by a user.
2. The application goes through the vector store and finds documents// texts similar to the prompt.
3. The application adds those documents to the prompt as context.
4. OpenAI returns a good response.

Btw, for the vector store you can use Turbine, it's a fully managed vector search engine that integrates with your database easily. https://useturbine.com/ Full disclosure, I'm one of the creators of Turbine. DM me if you wanna know more.

[karmic gyro]

So it’s not the model that decides to query the vector store but rather the app’s responsibility?

[heavy bluff]

yep

[karmic gyro]

Ok. I had hoped that I could use the embeddings endpoints to feed data into the system that the completion endpoints could then use on demand for my requests based on the model’s own judgement. Kind of how the plugins in ChatGPT work.

In particular how the Voxscript plugin is triggered automatically when I ask it to look something up.

[karmic gyro]

I guess the closest i could get in offloading logic to the model is to use function definitions that would indicate what kind of data can be found in the embedding store. Then the app only has to blindly react to function calls to fetch embeddings without having to understand the prompt.