Llama3 not supported ?

[safe orchid]

Hello,

I have started a server using lmstudio and llama3 model (same problem with any other model I've tested).
I'm chatting with the said model using autogen without problem.
Though, I followed the autogen Tool Use tutorial* and instead of using the registered function, the model ignores it and answers the question by itself.

I suppose I do something wrong but I don't find what.

• https://microsoft.github.io/autogen/docs/tutorial/tool-use

Here is the code. Any help to understand why this is not working would be greatly appreciated 🙂

import tempfile
from typing import Annotated, Literal
from autogen import ConversableAgent

assistant_system_message = """
You are a helpful AI assistant.
You can help with simple calculations.
Return 'TERMINATE' when the task is done.
"""

assistant = ConversableAgent(
    "Assistant",
    llm_config={
        "config_list": [
            {
                "model": "lmstudio-community/Meta-Llama-3-8B-Instruct-Q8_0",
                "base_url": "http://localhost:1234/v1",
                "api_key": "lm-studio"
            },
        ],
    },
    system_message=assistant_system_message,
)

user_proxy = ConversableAgent(
    name="User",
    llm_config=False,
    is_termination_msg=lambda x: x.get("content", "") and x.get("content", "").rstrip().endswith("TERMINATE"),
    human_input_mode="NEVER",
)

Operator = Literal["+", "-", "*", "/"]

@user_proxy.register_for_execution()
@assistant.register_for_llm(name="calculator", description="A simple calculator.")
def calculator(a: int, b: int, operator: Annotated[Operator, "operator"]) -> int:
    if operator == "+":
        return a + b
    elif operator == "-":
        return a - b
    elif operator == "*":
        return a * b
    elif operator == "/":
        return int(a / b)
    else:
        raise ValueError("Invalid operator")

chat_result = user_proxy.initiate_chat(assistant,message="What is (44232 + 13312 / (232 - 32)) * 5?")

[safe orchid]

It also looks like the summary is not working.

I have two agents talking. The first one ask to the second : "What is a pen?"
When I print the output of their chat, the summary section is... empty or incorret.
Exemple:

summary= "Do you have any specific questions or topics you'd like to delve into?"

Here is the small code I used.

from autogen import ConversableAgent
import pprint

teacher_agent = ConversableAgent(
    name="Teacher_Agent",
    system_message="You are a math teacher.",
    llm_config={
        "config_list": [
            {
                "model": "llama3",
                "base_url": "http://localhost:1234/v1",
                "api_key": "lm-studio"
            },
        ],
        "cache_seed": None,
    },
)
student_agent = ConversableAgent(
    name="Student_Agent",
    system_message="You are a student willing to learn.",
    llm_config={
        "config_list": [
            {
                "model": "llama3",
                "base_url": "http://localhost:1234/v1",
                "api_key": "lm-studio"
            },
        ],
        "cache_seed": None,
    },
)

chat_result = student_agent.initiate_chat(
    teacher_agent,
    message="What is a pen?",
    summary_method="reflection_with_llm",
    max_turns=2,
)

pprint.pprint(chat_result)

[safe orchid]

I just tested with an openai key and both tests do work.

So, the problem seems to be related to the lmstudio - autogen couple.

Any idea?

[safe orchid]

Humm...
The groupchat doesn't work neither with llama3.

Might be related to what is written in the doc of Llama3: A prompt should contain a single system message, can contain multiple alternating user and assistant messages, and always ends with the last user message followed by the assistant header.*
Which is the exact opposite of what autogen does for the group chat feature : one system message at the begining of the prompt and one other at the end.

So, long story short, it looks like autogen does not support llama3. 😭

*https://llama.meta.com/docs/model-cards-and-prompt-formats/meta-llama-3/

Llama3 not supported ?