#Inference Speed

1 messages · Page 1 of 1 (latest)

proven snow
#

When using the website for inference, after 3 prompts it gets so slow until it finally stops mid generation.
This is with all gpts.

Do you have any advice?

rotund edgeBOT
#

Hi @proven snow. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave:

#

Yes, I can provide some advice based on the principles of latency optimization outlined in the OpenAI documentation. Here are some strategies you might consider:

  1. Process tokens faster: The inference speed, which is the rate at which the language model processes tokens, is influenced by the model size. Smaller models usually run faster and cheaper, and when used correctly can even outperform larger models. You might want to consider using a smaller model if you're not already doing so. source (https://platform.openai.com/docs/guides/latency-optimization/1-process-tokens-faster)

  2. Generate fewer tokens: The number of tokens generated can also affect the speed of the model. You might want to adjust your settings to generate fewer tokens. source (https://platform.openai.com/docs/guides/latency-optimization/2-generate-fewer-tokens)

  3. Use fewer input tokens: The number of input tokens can also affect the speed of the model. You might want to adjust your settings to use fewer input tokens. source (https://platform.openai.com/docs/guides/latency-optimization/3-use-fewer-input-tokens)

  4. Make fewer requests: The number of requests made can also affect the speed of the model. You might want to adjust your settings to make fewer requests. source (https://platform.openai.com/docs/guides/latency-optimization/4-make-fewer-requests)

  5. Parallelize: If possible, you might want to consider parallelizing your requests to improve speed. source (https://platform.openai.com/docs/guides/latency-optimization/5-parallelize)

#

  1. Make your users wait less: This principle is more about perception than actual speed, but it's worth considering. You might want to look for ways to make the experience feel faster for your users, even if the actual speed doesn't change. source (https://platform.openai.com/docs/guides/latency-optimization/6-make-your-users-wait-less)

  2. Don't default to an LLM: Depending on your use case, you might want to consider not defaulting to a large language model (LLM). source (https://platform.openai.com/docs/guides/latency-optimization/7-don-t-default-to-an-llm)

Remember, these are just general principles and your specific situation might require a different approach. It's always a good idea to experiment and see what works best for you.

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

#

@proven snow kapa.ai is still learning and improving, please let us know how it did by reacting below