#8k Token Limit in ChatGPT Pro Impedes Complex Use-Cases and Lacks Input Truncation Alerts

1 messages · Page 1 of 1 (latest)

patent quarry
#

Executive Summary:

ChatGPT Pro is constrained by an 8k token limit, which proves insufficient for various complex use-cases. This limitation is particularly problematic when initial user inputs exceed the 8k token threshold, resulting in outputs of compromised integrity without any user notification of input truncation. The issue persists despite the existence of a 32k token model in ChatGPT Enterprise.

Expected Behavior:

  1. ChatGPT Pro should be capable of accommodating tasks that require a token count beyond the current 8k limit.
  2. The system should provide explicit user notifications if input exceeds the allowable token limit and is truncated as a result.

Observed Behavior:

  1. The 8k token limit restricts the platform's applicability for tasks requiring extended token counts.
  2. Exceeding this token limit in an initial user input leads to outputs of diminished integrity without any accompanying alert regarding input truncation.

Steps to Replicate:

  1. Engage ChatGPT Pro in tasks requiring more than 8k tokens.
  2. Encounter performance limitations upon reaching the 8k token threshold.
  3. Provide an initial input that surpasses the 8k token limit.
  4. Observe that the output is compromised and lacks any alert for input truncation.

Suggested Remedies:

  1. Enable an option to extend the token limit within ChatGPT Pro.
  2. Implement a user notification mechanism for instances where input truncation occurs due to token limitations.

Additional Context:

The current limitations in token count not only impede the range of feasible tasks but also lack transparency by failing to notify users when their inputs are truncated.

odd oasis
#

Token limit is not a bug, it's a technical limitation at the moment. While like you I would also love 32k token model from Enterprise, it's just not possible at the moment. It requires much larger allocation of compute at large scale. And my guess it only makes sense for OpenAI to do it with high paying corporate customers at large scale at the moment. If they released a personal Enterprise plan for $5,000 a month, it would be a PR disaster.

#

You can use 32k model via developer API, but it would cost you a lot even to process a small amount of information.

patent quarry
#

Reading is key

odd oasis
patent quarry
#

So it's not a bug that the form allows you to input more than 8k tokens, without informing you that the model never even saw the final 20% portion of your input?

#

Please stick to commenting on topics about which you are better informed

odd oasis