🔒 LanceDB cannot query

[still stirrup]

@lime halo

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

[lime halo]

def upload_to_lancedb(db_path, table_name, pdf_path):
    """Upload PDF text chunks and embeddings to LanceDB."""
    try:
        db = lancedb.connect(db_path)
        text = extract_text_from_pdf(pdf_path)
        
        if not text:
            raise ValueError("No text extracted from the PDF.")

        chunks = chunk_text_with_overlap(text, chunk_size=1000, overlap=20)
        embeddings = generate_embeddings_with_sentence_transformers(chunks)
        
        if len(embeddings) != len(chunks):
            raise ValueError("Mismatch between number of chunks and embeddings.")
        
        data = [{"id": i, "text": chunk, "embedding": embeddings[i]} for i, chunk in enumerate(chunks)]

        # Create or open the table and ensure the vector column is set correctly
        if table_name in db.table_names():
            table = db.open_table(table_name)
            print(f"Table '{table_name}' already exists. Using the existing table.")
        else:
            # Create the table with the 'embedding' column explicitly set as a vector
            table = db.create_table(table_name, data=data, vector_columns=["embedding"])
            print(f"Table '{table_name}' created and data uploaded.")

        print(f"Uploaded {len(data)} chunks from {pdf_path} to LanceDB.")
    except Exception as e:
        print(f"Error uploading to LanceDB: {e}")

def query_lancedb_rag(db_path, table_name, query_text):
    """Query LanceDB using the RAG model with Ollama."""
    try:
        db = lancedb.connect(db_path)
        table = db.open_table(table_name)
        
        model = SentenceTransformer('all-MiniLM-L6-v2')
        query_embedding = model.encode([query_text])[0]

        # Search for the most relevant chunks in LanceDB using the "embedding" column
        results = table.search(query_embedding, vector_column=["embedding"]).limit(5).to_pandas()
        context = "\n".join(results["text"])

        # Query Ollama with the context and query
        prompt = f"Context:\n{context}\n\nQuestion: {query_text}"
        response = query_ollama(prompt)

        if response:
            return response
        else:
            raise Exception("No response from Ollama.")
    
    except Exception as e:
        print(f"Error querying LanceDB or Ollama: {e}")
        return None


# Example usage
db_path = "lancedb"
table_name = "pdf_data"

Error querying LanceDB or Ollama: LanceTable.search() got an unexpected keyword argument 'vector_column'

how do I fix this?

if I remove vector_column=["embedding"] from anywhere it gives error saying missing or unexpected arg

[still stirrup]

@lime halo

Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.