Can you tell me how to apply streaming in chat completion api in nodejs
#Streaming in chat completion api
random sedge
robust slateBOT
Hi @random sedge. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave:
Sure, you can use the stream option in the chat.completions.create method of the OpenAI Node API library to get a stream of responses. Here's an example of how you might use it:
public async chat_completion(): Promise<any> {
const result: Stream<ChatCompletionChunk> = await this.ai.chat.completions.create({
max_tokens: this.options.max_tokens,
model: this.options.model,
messages: [
{
role: 'system',
content: this.options.system_prompt
},
...this.conversation
],
stream: this.options.stream as true
});
let finalMessage = '';
// transform the stream and concatenate response to finalMessage for internal use on my class
return modifiedStream;
}
In this example, this.ai.chat.completions.create is called with a stream option set to true. This returns a Stream of ChatCompletionChunk objects. You can then transform the stream as needed for your application.
Please note that you need to replace this.options.max_tokens, this.options.model, this.options.system_prompt, this.conversation, and this.options.stream with your actual values.
You can find more details in the source (https://github.com/openai/openai-node/issues/18).
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
random sedge
@robust slate in nodejs
robust slateBOT
Let me check the knowledge base for you. One moment! :books:
Sure, here's an example of how to use streaming with the OpenAI Node API library in Node.js:
console.log('Streaming:');
const stream = await openai.chat.completions.create({
model: 'gpt-3.5-turbo',
messages: [{ role: 'user', content: 'Say hello!' }],
stream: true,
});
for await (const part of stream) {
process.stdout.write(part.choices[0]?.delta?.content ?? '');
}
process.stdout.write('\n');
In this example, openai.chat.completions.create is called with a stream option set to true. This returns a stream of responses. The for await...of loop is used to asynchronously iterate over the stream and write each chunk of content to the standard output.
Please replace 'gpt-3.5-turbo' and 'Say hello!' with your actual model and message.
You can find more details in the source (https://github.com/openai/openai-node/issues/701).
Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:
random sedge
@robust slate i want to console log the streaming response
robust slateBOT
Let me check the knowledge base for you. One moment! :books:
Sure, you can use console.log to print the streaming response to the console. Here's how you can modify the previous example to do this:
console.log('Streaming:');
const stream = await openai.chat.completions.create({
model: 'gpt-3.5-turbo',
messages: [{ role: 'user', content: 'Say hello!' }],
stream: true,
});
for await (const part of stream) {
console.log(part.choices[0]?.delta?.content ?? '');
}
In this example, console.log is used instead of process.stdout.write to print each chunk of content to the console.
Please replace 'gpt-3.5-turbo' and 'Say hello!' with your actual model and message.