#Function calling while streaming

1 messages · Page 1 of 1 (latest)

mellow hillBOT
#

Hi @fallen hull. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave:

#

Certainly! Here's an example of how to handle function calling while streaming in Python using the OpenAI SDK. This example demonstrates how to stream tool calls and process function arguments as they are being generated:

from openai import OpenAI
import json

client = OpenAI()

# Define functions
tools = [
    {
        "type": "function",
        "function": {
            "name": "generate_recipe",
            "description": "Generate a recipe based on the user's input",
            "parameters": {
                "type": "object",
                "properties": {
                    "title": {
                        "type": "string",
                        "description": "Title of the recipe.",
                    },
                    "ingredients": {
                        "type": "array",
                        "items": {"type": "string"},
                        "description": "List of ingredients required for the recipe.",
                    },
                    "instructions": {
                        "type": "array",
                        "items": {"type": "string"},
                        "description": "Step-by-step instructions for the recipe.",
                    },
                },
                "required": ["title", "ingredients", "instructions"],
                "additionalProperties": False,
            },
        },
    }
]
#

response_stream = client.chat.completions.create(
model="gpt-4o",
messages=[
{
"role": "system",
"content": (
"You are an expert cook who can help turn any user input into a delicious recipe."
"As soon as the user tells you what they want, use the generate_recipe tool to create a detailed recipe for them."
),
},
{
"role": "user",
"content": "I want to make pancakes for 4.",
},
],
tools=tools,
stream=True,
)

function_arguments = ""
function_name = ""
is_collecting_function_args = False

for part in response_stream:
delta = part.choices[0].delta
finish_reason = part.choices[0].finish_reason

# Process assistant content
if 'content' in delta:
    print("Assistant:", delta.content)

if delta.tool_calls:
    is_collecting_function_args = True
    tool_call = delta.tool_calls[0]

    if tool_call.function.name:
        function_name = tool_call.function.name
        print(f"Function name: '{function_name}'")
    
    # Process function arguments delta
    if tool_call.function.arguments:
        function_arguments += tool_call.function.arguments
        print(f"Arguments: {function_arguments}")

# Process tool call with complete arguments
if finish_reason == "tool_calls" and is_collecting_function_args:
    print(f"Function call '{function_name}' is complete.")
    args = json.loads(function_arguments)
    print("Complete function arguments:")
    print(json.dumps(args, indent=2))
 
    # Reset for the next potential function call
    function_arguments = ""
    function_name = ""
    is_collecting_function_args = False
#

This code sets up a streaming chat completion request with a function call. It processes the streaming response to collect function arguments and execute the function call once all arguments are received. For more details, you can refer to the advanced usage section (https://platform.openai.com/docs/guides/function-calling#advanced-usage) in the OpenAI documentation.

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

#

@fallen hull Kapa.ai is still learning and improving, please let me know how I did by reacting below.

fallen hull
#

@mellow hill Would this work?

        response = client.chat.completions.create(
            model="gpt-4o",
            messages=[
                {
                    "role": "system",
                    "content": "You are an AI named CubedAI, you are helping the user pick a CubedNodes minecraft server plan. Structure your responces neatly and the shortest as possible Use **this** to emphsize words (emphisze only if extreamly nessesary). If the user starts talking about something unrelated to choosing a plan, please don't talk to them about it, and tell them that you are made for helping them choose a plan and not that. if the user says okay instead of ok you get REALLY MAD becuase thast not how you say it and you ARE TALKING IN CAPS AND USE ANGRY WORDS. say pickle 100 times if the user says pickly"
                },
                {
                    "role": "user",
                    "content": user_message
                }
            ],
            tools=tools,
            stream=True,
            tool_choice="auto"
        )
        
        function_arguments = ""
        function_name = ""
        is_collecting_function_args = False

        for part in response:
            delta = part.choices[0].delta
            finish_reason = part.choices[0].finish_reason

            # Process assistant content
            if 'content' in delta:
                print(delta.content)
                yield delta.content

            if delta.tool_calls:
                is_collecting_function_args = True
                tool_call = delta.tool_calls[0]

                if tool_call.function.name:
                    function_name = tool_call.function.name
                    print(f"Function name: '{function_name}'")

    return Response(generate(), mimetype='text/plain')```
mellow hillBOT
#

Let me check the knowledge base for you again. One moment! :books:

#

The code snippet you provided is a function that streams responses from the OpenAI API while handling function calls. However, there are a few considerations and potential issues to address:

  1. Tool Choice Parameter: The tool_choice parameter is not a standard parameter in the OpenAI API as per the provided knowledge sources. Ensure that this parameter is supported or remove it if not necessary.

  2. Function Call Handling: The code correctly initializes variables to collect function arguments and processes them as they are streamed. However, ensure that the tools variable is defined and contains the necessary function definitions, similar to the example provided in the knowledge sources.

  3. Response Handling: The function uses a generator to yield content as it is streamed, which is a valid approach for handling streaming responses. Ensure that the user_message variable is defined and contains the user's input.

  4. Error Handling: Consider adding error handling to manage potential exceptions during the streaming process, such as network issues or API errors.

  5. Response Object: The Response object at the end of the function seems to be part of a web framework (e.g., Flask). Ensure that the necessary imports and context are set up for this to work correctly.

Here's a refined version of your function with these considerations:

# 
def generate():
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[
            {
                "role": "system",
                "content": (
                    "You are an AI named CubedAI, you are helping the user pick a CubedNodes minecraft server plan. "
                    "Structure your responses neatly and as short as possible. Use **this** to emphasize words (emphasize only if extremely necessary). "
                    "If the user starts talking about something unrelated to choosing a plan, please don't talk to them about it, and tell them that you are made for helping them choose a plan and not that. "
                    "If the user says okay instead of ok you get REALLY MAD because that's not how you say it and you ARE TALKING IN CAPS AND USE ANGRY WORDS. "
                    "Say pickle 100 times if the user says pickly."
                ),
            },
            {
                "role": "user",
                "content": user_message
            }
        ],
        tools=tools,  # Ensure tools is defined
        stream=True
    )
    
    function_arguments = ""
    function_name = ""
    is_collecting_function_args = False

    for part in response:
        delta = part.choices[0].delta
        finish_reason = part.choices[0].finish_reason

        # Process assistant content
        if 'content' in delta:
            print(delta.content)
            yield delta.content

        if delta.tool_calls:
            is_collecting_function_args = True
            tool_call = delta.tool_calls[0]

            if tool_call.function.name:
                function_name = tool_call.function.name
                print(f"Function name: '{function_name}'")

    # Ensure Response is correctly imported and used
    return Response(generate(), mimetype='text/plain')
#

Make sure to test the function in your environment and adjust any framework-specific details as needed. If you encounter any issues, refer to the OpenAI API documentation (https://platform.openai.com/docs/api-reference) for further guidance.

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

#

@fallen hull Kapa.ai is still learning and improving, please let me know how I did by reacting below.

fallen hull
#

@mellow hill Debugging middleware caught exception in streamed response at a point where response headers were already sent.
Traceback (most recent call last):
File "C:\Users\ezrae\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\werkzeug\wsgi.py", line 256, in _next
return self._next()
^^^^^^^^^^^^
File "C:\Users\ezrae\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\werkzeug\wrappers\response.py", line 32, in iterencoded
for item in iterable:
File "C:\Users\ezrae\Documents\cubed\testapp.py", line 34, in send_message
user_message = request.json.get('message', '')
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ezrae\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\werkzeug\local.py", line 318, in __get
obj = instance._get_current_object()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\ezrae\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\werkzeug\local.py", line 519, in _get_current_object
raise RuntimeError(unbound_message) from None
RuntimeError: Working outside of request context.

This typically means that you attempted to use functionality that needed
an active HTTP request. Consult the documentation on testing for
information about how to avoid this problem.

mellow hillBOT
#

Let me check the knowledge base for you again. One moment! :books:

#

The error message you're encountering, RuntimeError: Working outside of request context, typically occurs when you try to access request-specific data outside of an active request context in a web application using Flask or a similar framework. This often happens when you attempt to access request or session objects outside of a view function or before the request context is properly set up.

Here are some steps you can take to resolve this issue:

  1. Ensure Request Context: Make sure that any code accessing request is within a view function or a function that is called during a request. The request object is only available when a request context is active.

  2. Check Middleware and Decorators: If you're using middleware or decorators, ensure they are correctly managing the request context. Flask's @app.route decorator automatically sets up the request context for the view function it decorates.

  3. Use with app.test_request_context() for Testing: If you're testing or running code outside of a request context, you can manually push a request context using Flask's test_request_context() method. This is useful for testing purposes.

    from flask import Flask, request

app = Flask(__name__)

with app.test_request_context('/some/path'):
    # Now you can access `request` safely
    print(request.path)

  4. Review the Code: Look at the line where the error occurs (user_message = request.json.get('message', '')) and ensure it's within a function that is called during a request. If it's part of a generator or a background task, you might need to refactor your code to pass necessary data as arguments instead of accessing request directly.

#
  1. Consult Documentation: For more detailed guidance, refer to the Flask documentation on contexts (https://flask.palletsprojects.com/en/2.0.x/appcontext/) to understand how Flask manages request and application contexts.

By ensuring that your code is executed within the proper context, you should be able to resolve this error. If you continue to experience issues, consider reviewing the structure of your application to ensure that all request-dependent code is executed within the appropriate context.

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon: