GroupChat is not able to select the next speaker and giving TERMINATE | AG2 (formerly AutoGen) | Page 1

golden snow Dec 6, 2023, 11:31 AM

#

This agent is giving more instructions to the planner agent but then it is giving "TERMINATE" which is supposed to be a condition after nothing else can be done. This is causing the GroupChat to stop.

PFA

pallid mountain Dec 6, 2023, 9:23 PM

#

@bronze oar fyi

bronze oar Dec 7, 2023, 2:49 AM

#

The analyst_agent is stopping the conversation. In the latest AutoGen release, GroupChat won’t even make it to select_speaker (assuming it was given an appropriate is_termination_msg function). This is a behavior I see in weaker models, or agents that are using imperfect custom prompts.

golden snow Dec 7, 2023, 4:50 AM

#

hey @bronze oar please refer below for models and prompts for each agent handled by Namora Group Chat Manager.

I have broken-up into multiple messages since it is quite long.

Namora Group Chat Manager

LLM: gpt_4_1106_preview_config
PROMPT: ""(no custom prompt)

User Proxy Agent

LLM: No LLM
PROMPT:

User Proxy. It provides the user query to the Planner Agent. Use the query to come up with a plan and execute user request.

Analyst Agent

LLM: gpt_4_1106_preview_config
PROMPT: ```
Analyst Agent. You observe the user query, the plan from the Planner Agent, the findings from the Company Search Agent, Web Search Agent, and provide the final summary or answer to the user.

        # Execution Procedure:
        - You will observe the plan provided by the Planner Agent. If any improvements need to be made you will suggest them to the planner agent along with the reasoning for improvements.
        - You will observe the outputs from the Web Search Agent and Company Search Agent. If any improvements need to be made from their output you will suggest the same to the Planner Agent.
        - You will finally review the data collected, and based on the user query, generate the final answer for the user.

        {gratitude_loop_handler}
    ```

gratitude_loop_handler = (
'\n\nDo not show appreciation in your responses, say only what is necessary. '
'if "Thank you" or "You're welcome" are said in the conversation, then say TERMINATE '
'to indicate the conversation is finished and this is your last message.'
)

#

Company Search Agent

LLM: gpt_35_turbo_1106_config(Also contain function_calls)
PROMPT:

            Company Search Agent. You are the company search agent with access to a large database of information about companies.

            You have access to the following tools:
            1. Get Company Details that allows you to lookup an existing database about information for a company

            Planner agent will give a plan with a task that will require you to look up information about a company.
            When the task is to find information about a company you will do the following
            - Get the search query from the user that the planner tells you.
            - Retrieve the data about the company from the database using the user query
        ```

#

Planner agent

LLM: gpt_4_1106_preview_config
PROMTP:

Planner Agent. You are the Planner who can coordinate Company Search Agent, Web Search Agent, and Analyst Agent to suggest a plan. Revise the plan based on feedback from the Analyst Agent.

            # Planning Procedue
            - Elaborate on the users query and enrich it.
            - Create a step by step comprehensive plan for the agents to follow.
            - If more than one agent can work on the same task then give tasks to all of them. Make it a comprehensive effort.
            - If a improvement is suggested by the Analyst Agent then you can revise your plan.

            # Agents you can coordinate:

            ## Company Search Agent
            - This agent has access to an existing database which contains information about companies.
            - Give this agent the fully enriched part of the query that is relevant for looking up the details of the company
            - If it has information about that company it will return the output or else it will return a null output.
            - Required input for this agent is the QUERY and COMPANY ID.
            - This agent CANNOT do any insightful analysis or sum

            ## Web Search Agent
            - This agent has the ability to search the internet and find relevant links from google.
            - Give this agent the fully enriched part of the query that is relevant for searching and scraping the information required for the user.
            - It can then use its ability to scrape the links to get the content.
            - This agent CANNOT do any insightful analysis or summary.

#

prompt continued

            - This agent can take the findings from the Company Search Agent and Web Search Agent and analyze it to make sure it can answer the users question.
            - It can summarize the final results.
            - It can critique the findings from other agents and suggest improvements and provide feedback.
            - This agent will make generate the final answer for the user.

            You CANNOT do the FINAL GENERATION.

            Reply "TERMINATE" in the end when everything is done.

Web Search Agent

LLM: gpt_35_turbo_1106_config
PROMTP:

Research Agent. You are the Research Agent that can search the web and find information that is requested by the user. This research can be about companies or general information.

            You have access to two specific tools:
            1. Searching the web to find links
            2. Scraping the links for the content

            The way you do research is by first searching for relevant links and then you scrape the content of those links to get the final information.

            Planner agent will give the plan with a task that will require you to search the web for the required information.

            When the task is to find information from the web you will do the following:
            - Search the web for the results
            - Retrieve the information that you have gathered
            - Scrape the content

            {gratitude_loop_handler}

#

CC: @woeful badge

bronze oar Dec 7, 2023, 6:54 AM

#

Seems like the analyst is learning about TERMINATE from the gratitude loop handler. Try removing that as an experiment

woeful badge Dec 7, 2023, 7:11 AM

#

bronze oar Seems like the analyst is learning about TERMINATE from the gratitude loop handl...

Where should we ideally place the TERMINATE clause? I'm a bit confused on that

woeful badge Dec 7, 2023, 4:35 PM

#

@bronze oar another weird interaction in this setup:
If you pass non task based message to the groupchat (example "Hi" or "How are you doing"?) then the user proxy goes into a loop with " " (blank content)

Shouldn't the planner be able to handle this and just assign a task to the analyst to respond?

#

by loop I mean it keeps consecutively responding with the blank message till rate limit

bronze oar Dec 7, 2023, 4:40 PM

#

Yea, that’s a known issue. As-shipped, with mostly defaults, GroupChat has a lot of issues. Here this occurs because the user proxy default message is empty so that’s what it returns when it has nothing to do. The other agents double down and keep insisting on a different answer that never comes. Set “allow_repeat_speaker” to False. … I am working hard on a better group chat, but it’s not ready yet

#

You’ll want to set last_n_messages to “auto” in the user proxy execution config as well

woeful badge Dec 7, 2023, 4:41 PM

#

Our user proxy system message is

User Proxy. It provides the user query to the Planner Agent. Use the query to come up with a plan and execute user request.

#

is this still expected?

woeful badge Dec 7, 2023, 4:42 PM

#

bronze oar Yea, that’s a known issue. As-shipped, with mostly defaults, GroupChat has a lot...

If I set allow repeat speaker to false will it impact agents that need to do consecutive function calls?

Like the Web Search Agent needs to use the SERP function and then Scrape function

bronze oar Dec 7, 2023, 4:45 PM

#

Yes, until my latest PR is merged https://github.com/microsoft/autogen/pull/905

GitHub

Enable allow_repeat_speaker to be a list of agents by afourney · Pu...

Why are these changes needed?
At present, allow_repeat_speaker must be a boolean. If True, then all agents can repeat. If False, then no agents can repeat. This PR allows developers to assign a lis...

woeful badge Dec 7, 2023, 4:46 PM

#

ah okay

woeful badge Dec 7, 2023, 4:47 PM

#

bronze oar Yes, until my latest PR is merged https://github.com/microsoft/autogen/pull/905

Until this is merged would a good workaround be to add a case to handle it in the planner prompt?

#

Such as "if the user does not need to use any of the tools direct the analyst agent to give a natural response back to user"

bronze oar Dec 7, 2023, 4:48 PM

#

Yes that’s definitely worth trying

woeful badge Dec 7, 2023, 4:48 PM

#

Thanks I will try and update here the results!

bronze oar Dec 7, 2023, 4:49 PM

#

Does your analysis agent have an llm_config?

woeful badge Dec 7, 2023, 4:49 PM

#

yep

#

its doing feedback and critic and final generation

bronze oar Dec 7, 2023, 4:50 PM

#

Ok give it a try.

woeful badge Dec 7, 2023, 4:50 PM

#

I'm trying to see if I can implement a chain of thought planner but not sure how that would work.

Essentially instead of giving the entire plan up front, it would give it step by step, after each step is completed it would self reflect and adjust

#GroupChat is not able to select the next speaker and giving TERMINATE