How to use tool calling? | OpenRouter | Page 1

vernal plaza Aug 1, 2024, 7:56 PM

#

I am providing tools, but none of the models that support tool calls actually return correct tool_use in the response?
I only see it in the content

I tried DeepSeek-Coder that supports it now
Tried Llama, mistral etc that show they support tool calls

What am I doing wrong

cobalt galleon Aug 1, 2024, 8:00 PM

#

See here for an example -> https://openrouter.ai/docs/requests#tool-calls

#

Or here for an actual call with return values -> #1265948512319967353 message

cobalt galleon Aug 1, 2024, 8:31 PM

#

Otherwise, feel free to post a tool call that does not work for you (best as a curl command) and we can debug it.

vernal plaza Aug 1, 2024, 10:40 PM

#

cobalt galleon Otherwise, feel free to post a tool call that does not work for you (best as a `...

~ ❯ curl https://openrouter.ai/api/v1/chat/completions
-H "Content-Type: application/json"
-H "Authorization: Bearer XXX"
-d '{
"model": "meta-llama/llama-3.1-70b-instruct",
"messages": [
{"role": "user", "content": "Read index.html"}
],
"tools":[
{
"type": "function",
"function": {
"name": "read_file",
"parameters": {
"type": "object",
"properties": {
"targetFile": {
"type": "string"
}
}
}
}
}],
}'

Response is:

{"id":"gen-gmIjxHcYgzuKYGKqsI4rPai9FmCe","model":"meta-llama/llama-3.1-70b-instruct","object":"chat.completion","created":1722551814,"choices":[{"index":0,"message":{"role":"assistant","content":"{\n "function": "read_file",\n "parameters": {\n "targetFile": "index.html"\n }\n}"},"finish_reason":"stop","logprobs":null}],"system_fingerprint":"601a0519fb4d41a706042f153a1732dce93cd158a93180a364be77ef4864bd39","usage":{"prompt_tokens":65,"completion_tokens":26,"total_tokens":91}}%

there is no tool_calls in message

Same for DeepSeek Coder, even when I set tool_choice to required

#

{"id":"gen-SNf20CuEzG3tVVBYES7D4usst9PG","model":"deepseek/deepseek-coder","object":"chat.completion","created":1722552247,"choices":[{"index":0,"message":{"role":"assistant","content":"json\n{\n \"name\": \"read_file\",\n \"parameters\": {\n \"targetFile\": \"index.html\"\n }\n}\n"},"finish_reason":"stop","logprobs":null}],"system_fingerprint":"fp_57bc6d0d10","usage":{"prompt_tokens":66,"completion_tokens":38,"total_tokens":104}}%

#

^ deepseek-coder wich should support tool calls in openai format but onl get content, no tool_calls

cobalt galleon Aug 1, 2024, 10:52 PM

#

vernal plaza ~ ❯ curl https://openrouter.ai/api/v1/chat/completions \ -H "Content-Type: app...

LLMs are still kind of dumb, you need to give the function tool schema something to work on in the description etc. Otherwise they will never know when to call those functions.
E.g. -> "description": "Get the current weather in a given location"
Also you need to tell the LLM that it actually has a choice to use the tool/function with "tool_choice": "auto",
Here is the fixed curl call ->

curl https://openrouter.ai/api/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $OPENROUTER_API_KEY" \
  -d '{
  "model": "meta-llama/llama-3.1-70b-instruct",
  "messages": [
    {"role": "user", "content": "Read index.html"}
  ],
      "tool_choice": "auto",
  "tools":[
    {
        "type": "function",
        "function": {
            "name": "read_file",
            "description": "Read a file from the filesystem and return its contents"
            "parameters": {
                "type": "object",
                "properties": {
                    "targetFile": {
                        "type": "string",
                        "description": "The file name as a string with extension e.g. page.html"
                    }
                }
            }
        }
    }],
}'

This returns call to a function, see ->

{"id":"gen-DxUdTYiF9D2iFDtVHlXnxRdqGOr2","model":"meta-llama/llama-3.1-70b-instruct","object":"chat.completion","created":1722552684,"choices":[{"index":0,"message":{"role":"assistant","content":"{\n  \"function\": \"read_file\",\n  \"parameters\": {\n    \"targetFile\": \"index.html\"\n  }\n}"},"finish_reason":"stop","logprobs":{"tokens":null,"token_logprobs":null,"top_logprobs":null,"text_offset":null}}],"usage":{"prompt_tokens":91,"completion_tokens":27,"total_tokens":118}}

vernal plaza Aug 1, 2024, 10:54 PM

#

^ in your response it is still in "content"

#

that needs to be in message.tool_calls not in message.content

cobalt galleon Aug 1, 2024, 10:56 PM

#

vernal plaza that needs to be in message.tool_calls not in message.content

Ah, see what you mean now. Here is the reply pretty printed for better visibility ->

{
  "id": "gen-ZZ01yjXj4DvBWv8ztCXA3lrLs06q",
  "model": "meta-llama/llama-3.1-70b-instruct",
  "object": "chat.completion",
  "created": 1722552934,
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "{\n  \"function\": \"read_file\",\n  \"parameters\": {\n    \"targetFile\": \"index.html\"\n  }\n}"
      },
      "finish_reason": "stop",
      "logprobs": null
    }
  ],
  "system_fingerprint": "601a0519fb4d41a706042f153a1732dce93cd158a93180a364be77ef4864bd39",
  "usage": {
    "prompt_tokens": 92,
    "completion_tokens": 26,
    "total_tokens": 118
  }
}

#

You correct that the function call in content will not really work.

vernal plaza Aug 1, 2024, 10:57 PM

#

right, it will be shown to user in chat, and will not be able to call tools

#

deepseek model actually should be working correctly based on their documentation
but same issue

cobalt galleon Aug 1, 2024, 10:58 PM

#

vernal plaza right, it will be shown to user in chat, and will not be able to call tools

The problem is, that this is not a fault of the model, but that the inference engine must support tool call / function calling and put these replies from the model in correct JSON

vernal plaza Aug 1, 2024, 10:59 PM

#

so is it possible to use any of the llama models and get correctly formatted response?

cobalt galleon Aug 1, 2024, 11:00 PM

#

vernal plaza so is it possible to use any of the llama models and get correctly formatted res...

If the inference engine is clever enough, yes.

#

BTW: The same exact call to openai/gpt-3.5-turbo-0125 produces a correct formatted JSON reply ->

{
  "id": "gen-cqkxlnQDTsshssieJoSaecMuYJPs",
  "model": "openai/gpt-3.5-turbo-0125",
  "object": "chat.completion",
  "created": 1722553180,
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": null,
        "tool_calls": [
          {
            "id": "call_t2Bfazcu09BQIv22sh5mcc0w",
            "type": "function",
            "function": {
              "name": "read_file",
              "arguments": "{\"targetFile\":\"index.html\"}"
            }
          }
        ]
      },
      "logprobs": null,
      "finish_reason": "tool_calls"
    }
  ],
  "system_fingerprint": null,
  "usage": {
    "prompt_tokens": 69,
    "completion_tokens": 16,
    "total_tokens": 85
  }
}

vernal plaza Aug 1, 2024, 11:01 PM

#

correct, openai models and claude models work for me

#

but I can't get any other models from OpenRouter to respond with tool calls 😦

#

I have some users that really want to use DeepSeek Coder or other models, and I can't get it to work 😦

#

with OpenRouter

cobalt galleon Aug 1, 2024, 11:02 PM

#

vernal plaza correct, openai models and claude models work for me

Tool use / function calling is still a bit like the wild west, no real standard and support is spotty, you have to test and see what works. Most often it is not the actual model that is the problem.

vernal plaza Aug 1, 2024, 11:03 PM

#

I tried DeepSeek coder directly through their API endpoint
when I require tool use, it responded correctly

However when using it through OpenRouter, I can't get it to work also
Even provider is DeepSeek

#

so I would think it would be the same

cobalt galleon Aug 1, 2024, 11:05 PM

#

vernal plaza so I would think it would be the same

Yes, that should work in theory. Dunno why it is not. Maybe @surreal turtle knows more?

vernal plaza Aug 1, 2024, 11:07 PM

#

Hey look:

"prompt_tokens": 69,

great job @cobalt galleon :))

sacred maple Aug 1, 2024, 11:12 PM

#

I think it might be because deepseek just recently added tool calling and openrouter hasn't updated for it.

Also for llama 3.1, if you look at meta's way of doing it, they just extract the tool result from the content part https://github.com/meta-llama/llama-agentic-system/blob/main/llama_agentic_system/system_prompt.py#L71

While we're here, if you stream tool calls, some models will give you malformed outputs https://discord.com/channels/1091220969173028894/1265886341648875522

GitHub

llama-agentic-system/llama_agentic_system/system_prompt.py at main ...

Agentic components of the Llama Stack APIs. Contribute to meta-llama/llama-agentic-system development by creating an account on GitHub.

vernal plaza Aug 1, 2024, 11:14 PM

#

sacred maple I think it might be because deepseek just recently added tool calling and openro...

Right I can try doing this:

Think very carefully before calling functions.
If you choose to call a function ONLY reply in the following format with no prefix or suffix:

<function=example_function_name>{{"example_name": "example_value"}}</function>

Was hoping OpenRouter helps make it universal where I don't have to do this per model

#

Was hoping I was doing something wrong and it would be a quick fix :))

Thanks for your help!

cobalt galleon Aug 1, 2024, 11:18 PM

#

vernal plaza Was hoping I was doing something wrong and it would be a quick fix :)) Thanks ...

I thought so too and, umm, was I wrong. Thanks for the hints @sacred maple 🙂

surreal turtle Aug 2, 2024, 8:30 AM

#

Looking soon, thanks all for the flag!

#

Yeah tool call is new on deepseek model, they also added FIM completion

surreal turtle Aug 2, 2024, 9:32 AM

#

tool call has been added to deepseek provider!

#How to use tool calling?