#Multiple vector indexes on the same table

9 messages · Page 1 of 1 (latest)

peak sky
#

I have a bunch of embeddings stored in a table. I want to be able to filter those on different columns depending on the scenario. I assume I'm meant to create two different vectorIndex's. But I keep getting the error when I try:

400 Bad Request: IndexFieldsNotUnique: Hit an error while evaluating your schema:
Index fields must be unique within an index.

The indexes I have on that table are:

        .vectorIndex("by_embedding", {
            vectorField: "embedding",
            dimensions: 1536,
            filterFields: ["clerkUserId"],
        })
        .vectorIndex("by_embedding_doc", {
            vectorField: "embedding",
            dimensions: 1536,
            filterFields: ["documentId"],
        }),

If I remove the second vectorIndex then it works. Any ideas what I'm doing wrong?

runic thorn
#

Interesting, this looks to me like it should work, so we'll look into it. But it also looks like you could use a single index .vectorIndex("by_embedding", {vectorField: "embedding", dimensions: 1536, filterFields: ["clerkUserId", "documentId"]}). For each vector search query you can filter on any subset of the filter fields.

strange sail
#

To expand on Lee's point a bit, a vectorIndex is different from a normal index in the way that filterFields work. You don't have to specify them in prefix order. You can filter on just documentId or just clerkUserId. As a result, you don't benefit from having two indexes on the same field. Of course let us know if you have a use case that would benefit from multiple indexes that I haven't thought of

peak sky
#

So specifying both fields in the one index ["clerkUserId", "documentId"] seems to work, thanks!

This mostly meets my needs - the only thing I'm missing is the ability to do ANDs in vector searches (intellisense and the docs both seem to indicate it's not available). When I'm filtering by documentId I would have liked to be able to also filter by clerkUserId to make sure I haven't stuffed up and am only looking at documents for the current user 🙂

strange sail
#

Gotcha. For that case, you could check the results when they come back, which would require fetching the document for the id returned from the search

runic thorn
#

@deft kindle and @vivid robin looked into supporting ANDs for filters on vector search, and iirc we discovered that it's an open research problem (as long as you want any kind of guarantees about the accuracy or runtime)

#

If the research problem is solved, we'll work on supporting it 😛. Isn't it fun working on cutting edge ai stuff

peak sky
#

Yeah, and checking the results after the fact is fine for this use case - hopefully it has no effect 😉