outer thistle
How do i continue the response after the chat.completitions.create calls a function?
I call the completitions with
openai.chat.completitions.create({
model: '...',
messages: [...],
tool: [...],
tool_choice: 'auto',
})
and when the completions return me a tool_calls, it won't continue generate any message, for example (in my case) 'I generated you an image of ...'. How do i make this happen?
coarse foxBOT
Hi @outer thistle. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave:
To continue the conversation after a function call, you need to add a new message to the messages array in your openai.chat.completions.create call. This new message should have the role set to "tool", and it should include the tool_call_id from the function call response. The content of this message should be the response from the function call.
Here is an example of how you can do this, based on the information provided in the OpenAI API Document page (https://platform.openai.com/docs/guides/function-calling) and the Github Issue (https://github.com/openai/openai-node/issues/642):
// Assume toolCall is the tool call response from the model
let toolCall = ...;
// Assume functionName is the name of the function that was called
let functionName = ...;
// Assume functionResponse is the response from the function call
let functionResponse = ...;
// Add a new message to the conversation with the function response
messages.push({
tool_call_id: toolCall.id,
role: "tool",
name: functionName,
content: functionResponse,
});
// Now make a new call to openai.chat.completions.create with the updated messages array
const response = await openai.chat.completions.create({
model: "...",
messages: messages,
tools: tools,
tool_choice: "auto",
});
name field in the ChatCompletionToolMessageParam type is currently under discussion in an open Github issue (https://github.com/openai/openai-node/issues/642), so you may encounter a type error when using it. You may need to adjust your code based on the resolution of this issue.
outer thistle
@coarse fox i have another problem, on the first response, when i get a tool_calls, i call a function that generates me an image, but it's async. How do i continue at generating the second response while the image is generating, then adding to content of second response message the image_url of the image once it has finished generating?
coarse foxBOT
Let me check the knowledge base for you. One moment! :books:
To handle asynchronous operations like generating an image, you can use JavaScript's built-in Promise functionality. You can start the image generation process, then continue with generating the second response. Once the image generation is complete, you can then update the content of the second response message with the image URL.
Here's a general idea of how you can do this:
// Start the image generation process
let imageGenerationPromise = generateImage();
// Continue with generating the second response
let secondResponse = await openai.chat.completions.create({
model: "...",
messages: messages,
tools: tools,
tool_choice: "auto",
});
// Once the image generation is complete, update the content of the second response
imageGenerationPromise.then(imageUrl => {
secondResponse.messages.push({
role: "system",
content: {
type: "image_url",
image_url: {
"url": imageUrl
}
}
});
});
In this code, generateImage is a function that starts the image generation process and returns a Promise that resolves to the URL of the generated image. The then method is used to specify what should happen once the Promise resolves, which in this case is updating the content of the second response message with the image URL.
outer thistle
@coarse fox now add the streaming both for the first request and the second. Note that the first request can also not make a tool_calls and just stream normal messages, so keep that in mind. When streaming is enabled and you get a tool call, the tool call parameters will be streamed
coarse foxBOT
Let me check the knowledge base for you. One moment! :books:
To add streaming to your requests, you need to set stream: true in your openai.chat.completions.create calls. This will make the model start returning tokens as soon as they are available, instead of waiting for the full sequence of tokens to be generated. This can improve the user experience by showing partial progress, as mentioned in the OpenAI API Document page (https://platform.openai.com/docs/guides/production-best-practices).
When streaming is enabled and you get a tool call, the tool call parameters will be streamed. You can handle this by checking for tool calls in each streamed response and starting the image generation process if a tool call is present.
Here's a general idea of how you can do this:
// Start the chat completion with streaming enabled
let chatCompletion = openai.chat.completions.create({{
model: "...",
messages: messages,
tools: tools,
tool_choice: "auto",
stream: true,
}});
// Handle each streamed response
chatCompletion.on('data', async (data) => {{
// Check if there is a tool call
if (data.tool_calls && data.tool_calls.length > 0) {{
// Start the image generation process
let imageGenerationPromise = generateImage();
// Once the image generation is complete, update the content of the second response
imageGenerationPromise.then(imageUrl => {{
// Add a new message to the conversation with the image URL
messages.push({{
role: "system",
content: {{
type: "image_url",
image_url: {{
"url": imageUrl
}}
}}
}});
// Continue with generating the second response
let secondResponse = await openai.chat.completions.create({{
model: "...",
messages: messages,
tools: tools,
tool_choice: "auto",
stream: true,
}});
}});
}}
}});
In this code, generateImage is a function that starts the image generation process and returns a Promise that resolves to the URL of the generated image. The then method is used to specify what should happen once the Promise resolves, which in this case is updating the content of the second response message with the image URL.
outer thistle
@coarse fox Now implement what you've made here:
const generateImage = async () => {
try {
const openai = new OpenAI({
apiKey: settings.apiKey,
dangerouslyAllowBrowser: true
});
return await openai.images.generate({
model: 'dall-e-3',
prompt: 'a white siamese cat',
size: '1024x1024',
quality: 'standard',
n: 1
});
} catch (error) {
}
};
const generateReply = async () => {
try {
const openai = new OpenAI({
apiKey: settings.apiKey,
dangerouslyAllowBrowser: true
});
let imagePrompt = '';
state.value.messages[0] = {
role: 'system',
content: [{ type: 'text', text: settings.systemPrompt }]
};
const stream = await openai.chat.completions.create({
model: 'gpt-4-1106-preview',
messages: getTrunkatedMessages(),
max_tokens: 4096,
stream: true,
temperature: settings.temperature,
presence_penalty: settings.presencePenalty,
frequency_penalty: settings.frequencyPenalty,
tools: [
{
type: 'function',
function: {
name: 'generateImage',
description: 'Generates an image given a prompt using DALL-E-3',
parameters: {
type: 'object',
properties: {
prompt: {
type: 'string',
description: 'The description to generate the image, it has to be very detailed.'
}
},
required: ['prompt']
}
}
}
],
tool_choice: 'auto'
});
addMessage('assistant', '');
for await (const chunk of stream) {
if (chunk.choices[0].delta.content) {
addChunk(chunk.choices[0].delta.content);
}
}
} catch (error) {
}
};
coarse foxBOT
Let me check the knowledge base for you. One moment! :books:
Based on the code you've provided, you can modify the generateReply function to handle tool calls and start the image generation process when a tool call is present. Here's how you can do this:
const generateReply = async () => {
try {
const openai = new OpenAI({
apiKey: settings.apiKey,
dangerouslyAllowBrowser: true
});
let imagePrompt = '';
state.value.messages[0] = {
role: 'system',
content: [{ type: 'text', text: settings.systemPrompt }]
};
const stream = await openai.chat.completions.create({
model: 'gpt-4-1106-preview',
messages: getTrunkatedMessages(),
max_tokens: 4096,
stream: true,
temperature: settings.temperature,
presence_penalty: settings.presencePenalty,
frequency_penalty: settings.frequencyPenalty,
tools: [
{
type: 'function',
function: {
name: 'generateImage',
description: 'Generates an image given a prompt using DALL-E-3',
parameters: {
type: 'object',
properties: {
prompt: {
type: 'string',
description: 'The description to generate the image, it has to be very detailed.'
}
},
required: ['prompt']
}
}
}
],
tool_choice: 'auto'
});
addMessage('assistant', '');
for await (const chunk of stream) {
// Check if there is a tool call
if (chunk.tool_calls && chunk.tool_calls.length > 0) {
// Start the image generation process
let imageGenerationPromise = generateImage();// Once the image generation is complete, update the content of the second response
imageGenerationPromise.then(imageUrl => {
// Add a new message to the conversation with the image URL
state.value.messages.push({
role: "system",
content: [{
type: "image_url",
image_url: {
"url": imageUrl
}
}]
});
// Continue with generating the second response
generateReply();
});
}
if (chunk.choices[0].delta.content) {
addChunk(chunk.choices[0].delta.content);
}
}
} catch (error) {
}
};
In this code, `generateImage` is a function that starts the image generation process and returns a Promise that resolves to the URL of the generated image. The `then` method is used to specify what should happen once the Promise resolves, which in this case is updating the content of the second response message with the image URL and then continuing with generating the second response.
Please note that this is a general approach and you may need to adjust the code to fit your specific use case and handle potential errors.
----------------------
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
----------------------