deft socket
For my application I simply need a number as a response to the prompts. I'm not exactly sure what I should do to achieve this? Do I simply give the instructions with each API call, or is this what I make an assistant for, or something else?
Regardless of the method, how does it affect the token usage? Will the instructions be counted as tokens for each API call?
crimson mantleBOT
Hi @deft socket. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave:
To get a number as a response to the prompts, you can specify your instruction in a way that guides the model to generate a numerical output. This can be done either by providing the instruction with each API call or by using an assistant, depending on your specific use case.
For example, if you're using the API directly, your instruction might look something like this: "Given the following data, calculate the average..." or "Predict the next number in this sequence...". The model will then generate a response based on your instruction.
As for token usage, both the input and output tokens count towards the total tokens used in an API call. This includes the tokens in your instruction. For instance, if your API call used 10 tokens in the message input and you received 20 tokens in the message output, you would be billed for 30 tokens[^1^][^2^].
The token usage also affects how long your API call takes, as writing more tokens takes more time[^1^][^2^]. Moreover, the total tokens must be below the model’s maximum limit (4097 tokens for gpt-3.5-turbo)[^1^][^2^].
You can check the token usage by looking at the usage field in the API response (e.g., response**'usage']['total_tokens'])[^1^][^2^]. To see how many tokens are in a text string without making an API call, you can use OpenAI’s [tiktoken** (https://github.com/openai/tiktoken) Python library[^1^][^2^].
Remember, managing costs is an important aspect of using the API. You can reduce costs by using shorter prompts, fine-tuning models, or caching common user queries[^3^][^4^].
deft socket
@crimson mantle So it's not like I can create a GPT with a set of instructions that will be read once and then followed from that point on? So that I would not have to send those instructions with each API call? In ChatGPT you can create custom GPTs for example.
crimson mantleBOT
Let me check the knowledge base for you. One moment! :books:
Yes, you're correct. In the context of ChatGPT, OpenAI allows you to create custom versions of the model, known as GPTs[^1^][^2^][^3^][^4^][^5^][^6^]. These GPTs can be tailored for specific tasks or topics by combining instructions, knowledge, and capabilities[^1^][^2^][^3^][^4^][^5^][^6^].
However, it's important to note that these custom GPTs are exclusive to the ChatGPT interface[^7^]. If you want to integrate GPT into a different interface, using the API is your only option[^7^].
When using the API, the instructions need to be sent with each API call[^1^][^2^][^3^][^4^][^5^][^6^][^7^]. The instructions are part of the input and will count towards the total tokens used in an API call[^1^][^2^][^3^][^4^][^5^][^6^][^7^].
deft socket
@crimson mantle So I can't use assistants for it? Or would that even increase the cost maybe?
crimson mantleBOT
Let me check the knowledge base for you. One moment! :books:
Assistants, as described in the OpenAI documentation, are entities powered by large language models like GPT-4[^1^]. They operate based on the instructions embedded within the context window of the model[^1^]. Assistants can perform complex tasks like running code or retrieving information from a file[^1^]. They can also have access to OpenAI-hosted tools like Code Interpreter and Knowledge Retrieval[^2^].
However, it's important to note that the usage of OpenAI-hosted tools comes at an additional fee[^2^]. For example, Code Interpreter is charged at $0.03 per session[^2^]. If your Assistant calls Code Interpreter simultaneously in two different threads, two Code Interpreter sessions are created[^2^]. Each session is active by default for one hour[^2^].
In terms of cost, using Assistants can indeed be more expensive due to their cost per query, maximum context length, iterative, and delayed output[^3^]. As a developer, you would bear the cost if you make an API that costs you significant money for every use[^3^].
So, while Assistants can be used to perform complex tasks, they might not be the most cost-effective solution for your needs. It's recommended to carefully consider the cost implications before deciding to use Assistants.
deft socket
@crimson mantle Sounds like I need to use fine-tuning
crimson mantleBOT
Let me check the knowledge base for you. One moment! :books:
Yes, fine-tuning can be a good approach if you have a specific application in mind and want the model to consistently follow certain instructions or guidelines[^1^][^2^][^3^][^4^][^5^].
Fine-tuning allows you to customize a model for your application by training it on more examples than can fit in a prompt[^1^][^2^][^3^][^4^][^5^]. This can lead to higher quality results, token savings due to shorter prompts, and lower latency requests[^1^][^2^][^3^][^4^][^5^].
The fine-tuning process involves preparing and uploading training data, training a new fine-tuned model, evaluating results, and iterating if needed[^1^][^2^][^3^][^4^][^5^]. Once a model has been fine-tuned, you won't need to provide as many examples in the prompt[^1^][^2^][^3^][^4^][^5^].
Fine-tuning is currently available for the following models: gpt-3.5-turbo-1106 (recommended), gpt-3.5-turbo-0613, babbage-002, davinci-002, and gpt-4-0613 (experimental)[^1^][^2^][^3^][^4^][^5^].
However, it's important to note that fine-tuning requires a careful investment of time and effort[^1^][^2^][^3^][^4^][^5^]. It's recommended to first attempt to get good results with prompt engineering, prompt chaining, and function calling[^1^][^2^][^3^][^4^][^5^].
ebon locust
@crimson mantle what is the path of least resistance to create an AI that can read user manuals and answer questions from it very well.
crimson mantleBOT
Let me check the knowledge base for you. One moment! :books:
Creating an AI that can read user manuals and answer questions from it very well involves several steps. Here's a general approach:
Data Preparation: First, you need to convert your user manuals into a format that can be processed by the AI. This might involve converting PDFs into text, or scraping web pages to extract the relevant information.
Information Retrieval: Next, you can use techniques like semantic search to find relevant sections of the manual based on a user's question[^1^]. This involves creating embeddings for your content and the user query, and then retrieving the most related content as measured by cosine similarity[^1^].
Question Answering: Once you have the relevant sections, you can use a language model like GPT-3 or GPT-4 to generate a response[^1^][^2^][^3^][^4^][^5^]. You can give the model the relevant sections as context, and then ask it to generate an answer to the user's question[^1^][^2^][^3^][^4^][^5^].
Fine-Tuning: If the default behavior of the language model isn't sufficient, you can fine-tune it on a custom dataset[^1^][^2^][^3^][^4^][^5^]. This involves training the model on examples of questions and answers based on your user manuals[^1^][^2^][^3^][^4^][^5^].
Iterative Improvement: Finally, you should continuously evaluate the performance of your AI and make improvements as necessary. This might involve collecting feedback from users, adding more training data, or adjusting the parameters of your models[^1^][^2^][^3^][^4^][^5^].
Remember, creating an effective AI for this task requires a careful balance of data preparation, model selection, and iterative improvement. It's also important to consider the ethical implications of your AI, such as how it handles user data and privacy[^1^][^2^][^3^][^4^][^5^].