rag with os model

[vestal belfry]

not sure if anyone has done this but am trying to do rag with an open source model. I copied one of the autogen exampels but keep getting this. Code I am using is here which is just from ```python

config_list = [
{
"api_type": "open_ai",
"api_base": "http://localhost:1234/v1",
"api_key": "NULL"
}
]

llm_config = {
"timeout": 60,
"cache_seed": 42,
"config_list": config_list,
"temperature": 0,
}

autogen.ChatCompletion.start_logging()

termination_msg = lambda x: isinstance(x, dict) and "TERMINATE" == str(x.get("content", ""))[-9:].upper()

boss = autogen.UserProxyAgent(
name="Boss",
is_termination_msg=termination_msg,
human_input_mode="NEVER",
system_message="The boss who ask questions and give tasks.",
code_execution_config=False, # we don't want to execute code in this case.
default_auto_reply="Reply TERMINATE if the task is done.",
)

boss_aid = RetrieveUserProxyAgent(
name="Boss_Assistant",
is_termination_msg=termination_msg,
system_message="Assistant who has extra content retrieval power for solving difficult problems.",
human_input_mode="NEVER",
max_consecutive_auto_reply=3,
retrieve_config={
"task": "code",
"docs_path": "",
"chunk_token_size": 1000,
"model": llm_config,
"client": chromadb.PersistentClient(path="/tmp/chromadb"),
"collection_name": "groupchat",
"get_or_create": True,
},
code_execution_config=False, # we don't want to execute code in this case.
)

Appreciate any help in trying to do this

[silent rampart]

@subtle stirrup fyi

[vestal belfry]

i’m actually considering just doing it myself

since autogen is just a framework

i think it has to do with the retrieveuserproxy being defined for only openai models

its cool

[subtle stirrup]

Hi @vestal belfry , local llm is supported. There are some errors in your configuration. Could you try below:

config_list = [
    {
        "model": "your model name",
        "api_base": "http://localhost:1234/v1",
        "api_key": "NULL"
    }
]


    retrieve_config={
        "task": "code",
        "docs_path": None,
        "chunk_token_size": 1000,
        "model": <Put your model name here, such as gpt-3.5.-turbo>,
        "client": chromadb.PersistentClient(path="/tmp/chromadb"),
        "collection_name": "groupchat",
        "get_or_create": True,
    },

you may need to replace 'api_base' with 'base_url' if you're using pyautogen>=0.2.

Please let me know if it works for you, thanks.

[vestal belfry]

will try now

[vestal belfry]

so if im using mistral-instruct

from hugging face

i just put that

?

[subtle stirrup]

Yes

[vestal belfry]

same error

[subtle stirrup]

The overall design is that if you can call your local llm endpoint with openai-python package, it can be used in autogen.

Check you config_list

[vestal belfry]

im definitely just messing up something simple here

oh yeah

forgot

sorry

[subtle stirrup]

you need to update "your model name"

[vestal belfry]

mb

mb

[subtle stirrup]

Check you pyautogen version

you may need to replace 'api_base' with 'base_url' if you're using pyautogen>=0.2.

From the error message, you're using pyautogen >= 0.2

[vestal belfry]

ah yeah im on 0.2

so ill swap

Its working!

thank you so much @subtle stirrup

sorry for those hiccups