GPT4 does not manage to use understand and apply a prompt specifying number of characters

[simple heart]

Countless times, I have asked GPT4 to extend or shorten an answer to a certain number of characters without success. For example, rewrite Y text with 800 characters to 1200 characters. It will rewrite it to a text that might be 500 or 2000 characters (seems random how long it gets) but it will tell me that it did the task. When I then tell it that it was wrong, it apologises, tries again, and fails again. The same problem happens if I ask it to extend a text by for example 5%.

[gusty pecan]

One thing to keep in mind is that it's "only" a language model, and not a math model. This means that when you try to specify a certain character/word count, there's nothing within the operation of the model itself that can calculate or count in the background--its only function is to generate natural language.

So, while it may be able to do something simple like "Write a 6-word sentence" or "Show me 3 words with 7 letters," this is because these are both basic requests mathematically and linguistically. And while it's not necessarily "complicated" to ask for a character count between 800 and 1200, it can be when you don't have a calculator, which is why you're seeing it just try its best to get approximately close to your request.

Perhaps you could experiment with using a plugin that can actually count, and somehow get it to do several "checks" with the plugin along the way to get you closer to your requested lengths.

[winter pike]

We can not say it is simply a language model since, as such, it makes too many grammatical mistakes. Also, an essay is a language assignment that involves a definite character limit.

[gusty pecan]

I'm just talking about how the model works: it doesn't count, it only generates language.

[winter pike]

I understand. We're here because we like the result and we want it to be even better.

[gusty pecan]

Definitely! I just responded to the post because it's in #1070006915414900886 and not #1070006151938314300 - in other words: it's not a bug that it can't consistently produce text within specified character counts, it's just not how the model works. It would be a suggestion to say, for example, "It would be nice if the language model was integrated with a math model such that I could request text of a specific word/character count."

[simple heart]

I disagree. If it not able to fulfill the request, it should say that it can’t do this reliably instead of presenting it like it fulfill the request. That is obviously a bug.

[past spoke]

That is not a bug. OpenAI has set a character limit on how many words The chat is trained to write.

[past spoke]

Plus chatgpt sucks at math. They are trained on a limited range of subjects and other subjects.

[simple heart]

I am not talking about exceeding the character limit.

I might have phrased my description poorly but what you are saying only supports my argument: If ChatGPT has a set character limit, then it must be able to count characters - otherwise that would not be possible. What I am stating is that it repeatedly fails to count characters correctly.

[gusty pecan]

I just asked ChatGPT about it and it told me as much:

While I aim to help with all kinds of tasks, I should note that as an AI, I don't have the ability to precisely control the character count of my responses. I can attempt to provide a longer or shorter response based on your request, but I don't have a built-in mechanism to ensure a response of exactly a specific number of characters.
ChatGPT doesn't have a character limit, it has a context limit, which means that the model can refer to past tokens up to a certain point, but there's nothing that indicates that the model itself can count or track these tokens, it's simply limited by them.

Perhaps the model should be more liberal in announcing its incapabilities if and when we ask it to do things it can't really do! However, hallucinations are common in this phase of the technology, whether it hallucinates a "fact" that never happened or hallucinates that it's producing the requested character count, so we have to be sure we're not assuming the model is doing exactly what it says it is, or is always correct, even if it doesn't tell us itself.

[simple heart]

"Perhaps the model should be more liberal in announcing its incapabilities if and when we ask it to do things it can't really do!"

That was all I said from the beginning 🙂

[gusty pecan]

The beginning sounded to me like you expected it to be functionally possible (tried it "countless times," etc.), and that it was therefore a bug that it was failing the request. I'm saying that since it's not functionally possible, this is more akin to a suggestion to either upgrade the model so that it becomes possible, or a suggestion to change the model to more readily self-identity when it is asked to do impossible things. I admit I'm being somewhat pedantic!