No more context, will terminate.

[topaz lynx]

I'm running the RAG example notebook and I get the following error for all examples: "
doc_ids: [[]]
No more context, will terminate.
ragproxyagent (to assistant):

TERMINATE
"

I have the documents, and I have pointed the ragproxyagent to the documents in docs_path.

Any idea what could be going wrong? the documents folder is not empty there are many .md files inside.

[shrewd silo]

@keen moon

[keen moon]

Hi @topaz lynx , thank you for the feedback. Is docs_path a relative path or absolute path? If it's a relative path, the working directory needs to be the root folder of the docs_path.

[topaz lynx]

it's an absolute path

[keen moon]

Could you share your code and a few docs for me to reproducing?

Do you see any errors during creating the collection?

[keen moon]

Btw, there is a known bug in the retrieve_utils.py, you may need to update

        # upsert in batch of 40000
        for i in range(0, len(chunks), 40000):
            collection.upsert(
                documents=chunks[
                    i : i + 40000
                ],  # we handle tokenization, embedding, and indexing automatically. You can skip that and add your own embeddings as well
                ids=[f"doc_{i}" for i in range(i, i + 40000)],  # unique for each doc
            )
        collection.upsert(
            documents=chunks[i : len(chunks)],
            ids=[f"doc_{i}" for i in range(i, len(chunks))],  # unique for each doc
        )

to

        # Upsert in batch of 40000 or less if the total number of chunks is less than 40000
        for i in range(0, len(chunks), min(40000, len(chunks))):
            end_idx = i + min(40000, len(chunks) - i)
            collection.upsert(
                documents=chunks[i:end_idx],
                ids=[f"doc_{j}" for j in range(i, end_idx)],  # unique for each doc
            )

The fix PR is under review, will merge soon.

https://github.com/microsoft/autogen/pull/50

[topaz lynx]

Thank you, I will update my code. I was also running this on Colab, I will move to my local machine so I can debug better.

[keen moon]

This is fixed in the latest main branch, should be OK now.

[shrewd silo]

Shall we make a new release?

[keen moon]

Yes, but I suggest after PR #50 is merged. I've left one small suggestion, could you check if it's OK merge the current version?

https://github.com/microsoft/autogen/pull/50

[shrewd silo]

Would you like to bump the version and write a release note?

[keen moon]

Sure, I'll make a draft.

[shrewd silo]

I'm bumping the version in https://github.com/microsoft/autogen/pull/59

[keen moon]

I didn't notice that😅

[shrewd silo]

no worries. both are merged.

[keen moon]

Are you preparing the release note?

I need to go out for a while, I can do it later if you're not working on it.

[shrewd silo]

I'm not. Please do it when you have time.

[keen moon]

OK

[keen moon]

I made a draft here: https://github.com/microsoft/autogen/releases/edit/untagged-a37648d36ea269e75b59

[shrewd silo]

Would you like to add a link to the example in your note?

[keen moon]

Example of using PDF files? We don't have an example notebook for that yet.

[shrewd silo]

Got it. Would be wonderful to have one. Maybe the contributor had one. Or we can ask others to give it a try. There are folks working on tutorials on HuggingFace

[keen moon]

I tried it, we can have different files ( a.txt, b.md, c.pdf, e.html ... ) under the given docs_path and all the files with supported formats will be parsed and added to the vector db.

From the users' perspective, no API is changed.

[shrewd silo]

Great. We can release it, announce it and encourage people to try. There is a community project thread: https://discordapp.com/channels/1153072414184452236/1156976658419568641