#Invalid Argument (error 400) with Vertex AI issue with formatted output and Swarm handoff

1 messages · Page 1 of 1 (latest)

low stag
#

I'm getting the following error

InvalidArgument: 400 For controlled generation of only function calls (forced function calling), please set 'tool_config.function_calling_config.mode' field to ANY instead of populating 'response_mime_type' and 'response_schema' fields. For more details, see: https://cloud.google.com/vertex-ai/generative-ai/docs/multimodal/function-calling#tool-config

When the agents have a response_format and there is a shared_context declared.

I took the example in https://docs.ag2.ai/docs/user-guide/basic-concepts/orchestration/swarm
And declared a response format for the lesson_planner agent.

class LessonPlan(BaseModel):
    title: Annotated[str, "Lesson plan title"]
    learning_objective: Annotated[str, "Key learning objectives"]
    script: Annotated[str, "How to introduce the topic to the kids"]

and

lesson_planner_llm_config = llm_config.copy()
lesson_planner_llm_config["response_format"] = LessonPlan
planner_message = """You are a classroom lesson planner.
Given a topic, write a lesson plan for a fourth grade class.
If you are given revision feedback, update your lesson plan and record it.
"""

# 3. Our agents now have tools to use (functions above)
lesson_planner = ConversableAgent(
    name="planner_agent", llm_config=lesson_planner_llm_config, system_message=planner_message, functions=[record_plan]
)
AG2
Swarm - AG2
#

using gemini-2.0-flash

tidal minnow
#

@low stag my first assumption is that gemini doesn't support function calls + response format together.
But I'll soon take a deeper look.

low stag
#

I've used function calling and format with gemini separately, The problem arise in the swarm when providing both (format + function calls).

tidal minnow
low stag
tidal minnow
# low stag

As I mentioned previously, gemini does not support function calls and structured output combined.
In this example, lesson_planner has both tools (functions) and response_format:

print(lesson_planner.llm_config)

{'model': 'gemini-2.0-flash',
 'api_type': 'google',
 'response_format': __main__.LessonPlan,
 'tools': [{'type': 'function',
   'function': {'description': 'Record the lesson plan',
    'name': 'record_plan',
    'parameters': {'type': 'object',
     'properties': {'lesson_plan': {'type': 'string',
       'description': 'lesson_plan'}},
     'required': ['lesson_plan']}}}]}