How to use Fine-Tuning Chat Completion API in Typescript?

[unreal aurora]

I have fine-tuned model and would like to use in nodejs with typescript and would like some help with few typescript errors. Please fine my code below:

• Function to generate response:

  async generateContentFromFineTunedModel(messages: OpenAIMessageParams[], maxTokens: number = 1000): Promise<OpenAI.Chat.Completions.ChatCompletionMessage> {
    try {
      const res = await this.openai.chat.completions.create({
        model: appConfig.app.backend.baseFineTunedModel,
        messages,
        temperature: 1,
        max_tokens: maxTokens,
        top_p: 1,
        frequency_penalty: 0,
        presence_penalty: 0,
      });

      return res.choices[0].message;
    } catch (error) {
      throw new Error(error);
    }
  }

Function Interface:

import { ChatCompletionAssistantMessageParam, ChatCompletionSystemMessageParam, ChatCompletionUserMessageParam } from 'openai/resources';

export type OpenAIMessageParams = ChatCompletionSystemMessageParam | ChatCompletionUserMessageParam  | ChatCompletionAssistantMessageParam

  generateContentFromFineTunedModel(messages: OpenAIMessageParams[], maxTokens?: number): Promise<OpenAI.Chat.Completions.ChatCompletionMessage> 

Function usage:

        const testJSON = [{"role": "system", "content": "Act as teacher"},{"role": "user", "content": "what is vowel"}]

        const response = await this.openAI.generateContentFromFineTunedModel(testJSON)

Error I am getting:

Argument of type '{ role: string; content: string; }[]' is not assignable to parameter of type 'OpenAIMessageParams[]'.
  Type '{ role: string; content: string; }' is not assignable to type 'OpenAIMessageParams'.
    Type '{ role: string; content: string; }' is not assignable to type 'ChatCompletionAssistantMessageParam'.
      Types of property 'role' are incompatible.
        Type 'string' is not assignable to type '"assistant"'.ts(2345)

[west pivot]

Hi @unreal aurora. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave:

---

The error you're encountering is due to TypeScript's strict type checking. In your testJSON array, you're using a general string type for the role property, but TypeScript expects a specific string literal type, such as "assistant", "system", or "user".

To fix this, you can explicitly declare the type of testJSON as OpenAIMessageParams[]. Here's how you can modify your function usage:

const testJSON: OpenAIMessageParams[] = [
  { role: "system", content: "Act as teacher" },
  { role: "user", content: "what is vowel" }
];

const response = await this.openAI.generateContentFromFineTunedModel(testJSON);

This way, TypeScript knows that testJSON is an array of OpenAIMessageParams, and it will enforce the correct types for the role and content properties.

For more information on how to use the OpenAI API with TypeScript, you can refer to the OpenAI Community Forum (https://community.openai.com/t/443793) and the OpenAI GitHub Issues (https://github.com/openai/openai-node/issues/639) where similar topics are discussed.

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

@unreal aurora kapa.ai is still learning and improving, please let us know how it did by reacting below

[unreal aurora]

@west pivot
ok now i am getting content from gpt fine tuned models as below:

{
  role: 'assistant',
  content: '[\n' +
    '    "Shape the Future with Us",\n' +
    '    "Join Our Team Today",\n' +
    '    "Empower Your Career",\n' +
    '    "Create a Brighter Future",\n' +
    '    "Revolutionizing the Workplace",\n' +
    '    "Mark Your Impact Here",\n' +
    '    "Ready to Drive Change",\n' +
    '    "Lead the Way in There"    \n' +
    ']'
}

How do i validate it and format it properly as array and return to user?

[west pivot]

Let me check the knowledge base for you. One moment! :books:

---

The content you're receiving from the fine-tuned model is a string representation of a JSON array. You can parse this string into an actual array using JSON.parse() in JavaScript/TypeScript. Here's how you can do it:

const response = await this.openAI.generateContentFromFineTunedModel(testJSON);

// Parse the content string into an array
const contentArray = JSON.parse(response.content);

// Now contentArray is an actual array that you can manipulate or return to the user
console.log(contentArray);

This will output:

[
  "Shape the Future with Us",
  "Join Our Team Today",
  "Empower Your Career",
  "Create a Brighter Future",
  "Revolutionizing the Workplace",
  "Mark Your Impact Here",
  "Ready to Drive Change",
  "Lead the Way in There"
]

Please note that JSON.parse() can throw an error if the input string is not valid JSON. You might want to use a try-catch block to handle any potential errors.

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

@unreal aurora kapa.ai is still learning and improving, please let us know how it did by reacting below

[unreal aurora]

@west pivot I am also getting response like this:

{
  role: 'assistant',
  content: '[\n' +
    '    {\n' +
    '        headline: "Director of DevOps Innovation",\n' +
    '        jobRoleDescription: "We are seeking a visionary Director of DevOps Innovation to lead our technological transformation. In this role, you will be the driving force behind our DevOps initiatives, overseeing the development and implementation of cutting-edge solutions. Your role will involve formulating robust DevOps strategies, optimizing system performance, and ensuring seamless integration across all platforms. You will lead a talented team of DevOps engineers, fostering a culture of innovation and continuous improvement. As the Director of DevOps Innovation, you will be responsible for managing complex projects, guiding the team in agile methodologies, and liaising with key stakeholders to align technological advancements with business objectives. Your leadership will be instrumental in maintaining the highest standards of security, scalability, and reliability in our digital infrastructure, while also driving the development of cloud-native solutions. Bring a proven track record in leading DevOps initiatives, strong business acumen, and excellent leadership skills, coupled with a passion for staying abreast of the latest technological trends and driving organizational transformation through digital innovation."\n' +
    '    }\n' +
    ']'
}

Will this parse?

[west pivot]

Let me check the knowledge base for you. One moment! :books:

---

The content you're receiving in this case is a string representation of a JSON array containing an object. However, the object's properties (headline and jobRoleDescription) are not enclosed in quotes, which is required for valid JSON.

If the content string was valid JSON, you could parse it into an actual array of objects using JSON.parse(), similar to the previous example. However, in this case, JSON.parse() will throw an error due to the invalid JSON format.

Unfortunately, without modifying the source that generates this content to produce valid JSON, there's no reliable way to parse this string into an object. You might want to check the configuration of your fine-tuned model or the prompts you're using to ensure they're producing valid JSON output.

If you have control over the generation of this content, ensure that the properties of the objects are enclosed in quotes, like so:

[
  {
    "headline": "Director of DevOps Innovation",
    "jobRoleDescription": "We are seeking a visionary Director of DevOps Innovation to lead our technological transformation..."
  }
]

This is valid JSON and can be parsed with JSON.parse().

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

@unreal aurora kapa.ai is still learning and improving, please let us know how it did by reacting below

[unreal aurora]

@west pivot Yes but the fine-tuned model won't generate the content I want every time accurately can we write a function which handles this responses and gives us the generated oputput?

[west pivot]

Let me check the knowledge base for you. One moment! :books:

---

Yes, you can write a function to handle the responses from the fine-tuned model and format the output. However, due to the nature of the content string you're receiving, you'll need to handle two cases: when the content is a JSON array of strings, and when it's a JSON array of objects.

Here's a function that attempts to parse the content string into an array and handles any errors that might occur due to invalid JSON:

function parseContent(content: string): any[] | string {
  try {
    // Try to parse the content string into an array
    return JSON.parse(content);
  } catch (error) {
    // If an error occurred, the content string is not valid JSON
    // In this case, return the original content string
    return content;
  }
}

const response = await this.openAI.generateContentFromFineTunedModel(testJSON);

// Use the parseContent function to parse the content string
const parsedContent = parseContent(response.content);

// Now parsedContent is either an array (if the content string was valid JSON)
// or the original content string (if the content string was not valid JSON)
console.log(parsedContent);

This function will return the parsed array if the content string is valid JSON, or the original content string if it's not. This way, you can handle the output from the fine-tuned model in a more flexible way.

Please note that this function assumes that the content string, if it's valid JSON, represents an array. If the content string could represent other types of JSON (like an object or a primitive value), you might need to adjust the function accordingly.

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

@unreal aurora kapa.ai is still learning and improving, please let us know how it did by reacting below

[unreal aurora]

@west pivot Actually now I am getting response like this:

@west pivot
Actually, now I am getting response like this:

[west pivot]

Let me check the knowledge base for you. One moment! :books:

---

I'm sorry, but it seems like your question got cut off. Could you please provide more details or clarify what kind of response you're getting now? I'll be glad to help you further.

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

@unreal aurora kapa.ai is still learning and improving, please let us know how it did by reacting below

[unreal aurora]

{
id: 'chatcmpl-99GzMXbNjClNCWOMzZM7xb8VnF1QY',
object: 'chat.completion',
created: 1711997284,
model: 'ft:gpt-3.5-turbo-1106:write-recruit:wr-ft-c1-t13:985M1poe',
choices: [
{
index: 0,
message: [Object],
logprobs: null,
finish_reason: 'length'
}
],
@west pivot What does finish_reason mean here?

[west pivot]

Let me check the knowledge base for you. One moment! :books:

---

The finish_reason field in the response from the OpenAI API indicates why the model stopped generating further tokens. In your case, the finish_reason is "length", which means the model stopped generating further tokens because it hit the maximum token limit. This limit could be due to the max_tokens parameter you set in the API request or the inherent token limit of the model.

There are two possible values for finish_reason:

• "stop": The model stopped because it generated a message that was complete, or it encountered one of the stop sequences provided via the stop parameter in the API request.
• "length": The model stopped because it hit the maximum token limit.

For more information, you can refer to the OpenAI API documentation (https://platform.openai.com/docs/guides/text-generation).

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

@unreal aurora kapa.ai is still learning and improving, please let us know how it did by reacting below

[unreal aurora]

Ok I am using the API Like this:

const res = await this.openai.chat.completions.create({
        model: appConfig.app.backend.baseFineTunedModel,
        messages,
        temperature: 0.5,
        max_tokens: maxTokens,
        top_p: 1,
        frequency_penalty: 0,
        presence_penalty: 0,
      });

Can you provide how can i generate the remaining response?

@west pivot
Ok I am using the API Like this:
const res = await this.openai.chat.completions.create({
model: appConfig.app.backend.baseFineTunedModel,
messages,
temperature: 0.5,
max_tokens: maxTokens,
top_p: 1,
frequency_penalty: 0,
presence_penalty: 0,
});

Can you provide how can i generate the remaining response?

[west pivot]

Let me check the knowledge base for you. One moment! :books:

---

If the model's response is cut off because it hits the max_tokens limit, you can generate the remaining response by taking the last part of the previous response and using it as a prompt for a new API call. However, please note that this might not always result in a coherent continuation, as the model does not have a memory of past requests.

Here's a basic example of how you can do this:

let remainingTokens = maxTokens;
let messages = initialMessages;
let fullResponse = '';

while (remainingTokens > 0) {
  const res = await this.openai.chat.completions.create({
    model: appConfig.app.backend.baseFineTunedModel,
    messages,
    temperature: 0.5,
    max_tokens: remainingTokens,
    top_p: 1,
    frequency_penalty: 0,
    presence_penalty: 0,
  });

  // Append the new message to the full response
  fullResponse += res.choices[0].message.content;

  // Subtract the number of tokens in the new message from the remaining tokens
  remainingTokens -= res.choices[0].message.content.split(' ').length;

  // Use the new message as the prompt for the next API call
  messages = [{ role: 'user', content: res.choices[0].message.content }];
}

console.log(fullResponse);

In this example, initialMessages is the initial array of messages you're using as a prompt. The while loop continues to make API calls until it runs out of tokens. After each API call, it subtracts the number of tokens in the new message from the remaining tokens and uses the new message as the prompt for the next API call.

Please note that this is a simplified example and might not work perfectly for all use cases. For example, it assumes that the content of each message is a string of space-separated tokens, which might not be accurate if the content includes punctuation or other non-space separators. You might need to adjust the code to better fit your specific use case.

For more information on how to use the OpenAI API, you can refer to the OpenAI API documentation (https://platform.openai.com/docs/guides/text-generation).

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

@unreal aurora kapa.ai is still learning and improving, please let us know how it did by reacting below

[unreal aurora]

@west pivot But will it get a context of previous text being generated is left and wht needs to be generated furter?

[west pivot]

Let me check the knowledge base for you. One moment! :books:

---

The GPT-3 model, including its fine-tuned versions, does not have a built-in memory of past requests. Each API call is stateless, meaning it doesn't know about the previous calls. The only context it has for generating a response is the prompt you provide in the current API call.

In the example I provided, the last part of the previous response is used as the prompt for the next API call. This gives the model some context for generating a continuation. However, because the model doesn't remember the full context of the previous text, the continuation might not always be coherent or relevant.

If you need the model to have a full context of the conversation, you would need to include the entire conversation history in the messages parameter for each API call. This way, the model has all the necessary context to generate a relevant response. But please note that the total number of tokens in the messages and the response must be under the model's maximum limit (4096 tokens for gpt-3.5-turbo).

Here's an example:

const conversationHistory = [
  { role: 'system', content: 'You are a helpful assistant.' },
  { role: 'user', content: 'Who won the world series in 2020?' },
  // Include more messages as needed
];

const res = await this.openai.chat.completions.create({
  model: appConfig.app.backend.baseFineTunedModel,
  messages: conversationHistory,
  temperature: 0.5,
  max_tokens: maxTokens,
  top_p: 1,
  frequency_penalty: 0,
  presence_penalty: 0,
});

In this example, conversationHistory contains the entire conversation history. The model uses this history as context for generating its response.

For more information on how to use the OpenAI API, you can refer to the OpenAI API documentation (https://platform.openai.com/docs/guides/text-generation).

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

@unreal aurora kapa.ai is still learning and improving, please let us know how it did by reacting below

[unreal aurora]

@west pivot
I have tried it but not generating the pending incomplete response

[west pivot]

Let me check the knowledge base for you. One moment! :books:

---

If the model is not generating the remaining incomplete response as expected, it could be due to several reasons:

1. Token Limit: The model has a maximum token limit (4096 tokens for gpt-3.5-turbo). If the conversation history and the generated response exceed this limit, the model might not be able to generate a complete response.

2. Prompt Content: The model generates responses based on the prompt you provide. If the prompt does not clearly indicate that a continuation is needed, the model might not generate the expected response.

3. Model Limitations: The model, even though it's powerful, has its limitations. It might not always generate the expected response, especially for complex or ambiguous prompts.

Without more specific details about the prompts you're using and the responses you're getting, it's hard to provide a more precise answer. However, here are a few general tips:

• Make sure the prompt clearly indicates that a continuation is needed. For example, if the previous response ended in the middle of a sentence, include the incomplete sentence in the prompt.
• If the conversation history is long, try to summarize it or focus on the most relevant parts to avoid exceeding the token limit.
• Experiment with different temperature and max tokens settings. A higher temperature makes the output more random, while a lower temperature makes it more deterministic. Adjusting these parameters might help get a more complete response.

For more information on how to use the OpenAI API, you can refer to the OpenAI API documentation (https://platform.openai.com/docs/guides/text-generation).

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

@unreal aurora kapa.ai is still learning and improving, please let us know how it did by reacting below

[unreal aurora]

Token limits depend on the model you select. For gpt-3.5-turbo-0125, the maximum context length is 16,385 so each training example is also limited to 16,385 tokens

Is this limit per example?

@west pivot

[west pivot]

Let me check the knowledge base for you. One moment! :books:

Sorry @unreal aurora, I seem to be having a bit of trouble. Don't worry though, I've pinged my support and they will try to fix me up! :technologist:

Perhaps try again in 15mins?

[unreal aurora]

@west pivot Token limits depend on the model you select. For gpt-3.5-turbo-0125, the maximum context length is 16,385 so each training example is also limited to 16,385 tokens

Is this limit per example?

[west pivot]

Let me check the knowledge base for you. One moment! :books:

---

Yes, the token limit is per example. For the gpt-3.5-turbo-0125 model, each training example is limited to 16,385 tokens. This means that the total number of tokens in the input and output of a single API call cannot exceed 16,385.

However, it's important to note that the maximum number of tokens that can be generated in a single output is 4,096. So, if you're using the API to generate a response, you need to ensure that the total number of tokens in your input message(s) and the expected output does not exceed this limit.

For example, if your input message(s) contain 1,000 tokens, you can generate up to 3,096 tokens in the output. If your input message(s) contain 12,385 tokens, you can generate up to 4,000 tokens in the output, but not more, because that would exceed the total limit of 16,385 tokens.

You can refer to the OpenAI documentation (https://platform.openai.com/docs/guides/fine-tuning) and the OpenAI Community Forum (https://community.openai.com/t/565125) for more information on token limits.

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

@unreal aurora kapa.ai is still learning and improving, please let us know how it did by reacting below

[unreal aurora]

@west pivot If i am fine tuning my fine tuned model then what iss the token limit for it?

@west pivot If i am fine tuning my fine tuned model then what iss the token limit for it?

[west pivot]

Let me check the knowledge base for you. One moment! :books:

---

The token limit for fine-tuning a model depends on the base model you're using. For example, if you're fine-tuning a model based on gpt-3.5-turbo-0125, each training example is limited to 16,385 tokens. If you're using gpt-3.5-turbo-0613, each training example is limited to 4,096 tokens.

However, it's important to note that these limits apply to each individual training example, not the entire dataset. The total size of your dataset can be much larger.

When you're fine-tuning a fine-tuned model (i.e., using a fine-tuned model as the base model for further fine-tuning), the token limits would be the same as those of the base fine-tuned model.

You can refer to the OpenAI documentation (https://platform.openai.com/docs/guides/fine-tuning) and the OpenAI Community Forum (https://community.openai.com/t/563147) for more information on token limits for fine-tuning.

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

@unreal aurora kapa.ai is still learning and improving, please let us know how it did by reacting below

[unreal aurora]

@west pivot the mean if i initially made fine tune model on gpt-3.5-turbo-0125 and then if i again fine tune my fine tune model then the token for it is 16385

[west pivot]

Let me check the knowledge base for you. One moment! :books:

---

Yes, that's correct. If you initially fine-tuned a model based on gpt-3.5-turbo-0125, which has a token limit of 16,385 per training example, and then you fine-tune this fine-tuned model again, the token limit per training example would still be 16,385.

This is because the token limit is determined by the base model you're using for fine-tuning. In this case, the base model is gpt-3.5-turbo-0125, so the token limit is 16,385, regardless of whether you're fine-tuning the base model or a model that was previously fine-tuned on the base model.

For more information, you can refer to the OpenAI documentation (https://platform.openai.com/docs/guides/fine-tuning) and the OpenAI Community Forum (https://community.openai.com/t/563147).

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

@unreal aurora kapa.ai is still learning and improving, please let us know how it did by reacting below

[unreal aurora]

per the below dataset right?
{"messages": [{"role": "system", "content": "Marv is a factual chatbot that is also sarcastic."}, {"role": "user", "content": "What's the capital of France?"}, {"role": "assistant", "content": "Paris, as if everyone doesn't know that already."}]}

@west pivot per the below dataset right?
{"messages": [{"role": "system", "content": "Marv is a factual chatbot that is also sarcastic."}, {"role": "user", "content": "What's the capital of France?"}, {"role": "assistant", "content": "Paris, as if everyone doesn't know that already."}]}

[west pivot]

Let me check the knowledge base for you. One moment! :books:

---

Yes, that's correct. The token limit applies to each individual example in your dataset. In the context of fine-tuning a chat model, an example is a conversation that consists of a series of messages. Each message has a role (either "system", "user", or "assistant") and content.

Here's an example of a conversation:

{
  "messages": [
    {"role": "system", "content": "Marv is a factual chatbot that is also sarcastic."},
    {"role": "user", "content": "What's the capital of France?"},
    {"role": "assistant", "content": "Paris, as if everyone doesn't know that already."}
  ]
}

In this example, the total number of tokens is the sum of the tokens in each message's content. The token limit (e.g., 16,385 for gpt-3.5-turbo-0125) applies to this total number of tokens.

For more information, you can refer to the OpenAI documentation (https://platform.openai.com/docs/guides/fine-tuning) and the OpenAI Community Forum (https://community.openai.com/t/563147).

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

[unreal aurora]

@west pivot Is there any limit in number of datasets?

[west pivot]

Let me check the knowledge base for you. One moment! :books:

Sorry @unreal aurora, I seem to be having a bit of trouble. Don't worry though, I've pinged my support and they will try to fix me up! :technologist:

Perhaps try again in 15mins?

[unreal aurora]

@west pivot Is there any limit in number of datasets?

[west pivot]

Let me check the knowledge base for you. One moment! :books:

Sorry @unreal aurora, I seem to be having a bit of trouble. Don't worry though, I've pinged my support and they will try to fix me up! :technologist:

Perhaps try again in 15mins?

[unreal aurora]

@west pivot Is there any limit in number of datasets?

[west pivot]

Let me check the knowledge base for you. One moment! :books:

Sorry @unreal aurora, I seem to be having a bit of trouble. Don't worry though, I've pinged my support and they will try to fix me up! :technologist:

Perhaps try again in 15mins?

[unreal aurora]

@west pivot Is there any limit in number of datasets?

[west pivot]

Let me check the knowledge base for you. One moment! :books:

Sorry @unreal aurora, I seem to be having a bit of trouble. Don't worry though, I've pinged my support and they will try to fix me up! :technologist:

Perhaps try again in 15mins?

[unreal aurora]

@west pivot Is there any limit in number of datasets?

[west pivot]

Let me check the knowledge base for you. One moment! :books:

Sorry @unreal aurora, I seem to be having a bit of trouble. Don't worry though, I've pinged my support and they will try to fix me up! :technologist:

Perhaps try again in 15mins?

[unreal aurora]

@west pivot

[west pivot]

Let me check the knowledge base for you. One moment! :books:

Sorry @unreal aurora, I seem to be having a bit of trouble. Don't worry though, I've pinged my support and they will try to fix me up! :technologist:

Perhaps try again in 15mins?

[unreal aurora]

@west pivot Is there any limit in number of datasets?

[west pivot]

Let me check the knowledge base for you. One moment! :books:

Sorry @unreal aurora, I seem to be having a bit of trouble. Don't worry though, I've pinged my support and they will try to fix me up! :technologist:

Perhaps try again in 15mins?

[unreal aurora]

@west pivot Is there any limit in number of datasets?

[west pivot]

Let me check the knowledge base for you. One moment! :books:

Sorry @unreal aurora, I seem to be having a bit of trouble. Don't worry though, I've pinged my support and they will try to fix me up! :technologist:

Perhaps try again in 15mins?

[unreal aurora]

@west pivot Is there any limit in number of datasets?

[west pivot]

Let me check the knowledge base for you. One moment! :books:

Sorry @unreal aurora, I seem to be having a bit of trouble. Don't worry though, I've pinged my support and they will try to fix me up! :technologist:

Perhaps try again in 15mins?

[unreal aurora]

@west pivot Is there any limit in number of datasets?

[west pivot]

Let me check the knowledge base for you. One moment! :books: