GPT-4 in ChatGPT losing its memory after 1 message

[gusty sparrow]

Hi team, after 1-2 messages, GPT-4 inside of ChatGPT seems to be losing its memory after 1-2 messages and cannot comprehend the context of the conversation or task at hand. Is anyone else experiencing this?

[final blaze]

Share the conversation

[opal berry]

Had the same issue last night actually. GPT literally couldn't analyze the code in full. Kept telling me I gave it only snippets of code, but that was just because it would only analyze it in snippets xD

was at about 2am last night also

[modest spire]

This is happening for me as well.

I have been experimenting and I also noticed that when pasting a particularly long piece of code, GPT will respond with "It looks like your message was cut off, so I'm not sure what specific question or problem you might be facing."

I ask GPT what the last line received was, and I get the following:

[gusty sparrow]

https://chat.openai.com/share/46cde1c4-3c9b-45b9-acaf-5ca33f8a0fc5

it's any conversation with GPT-4, not just this one @final blaze

[gusty sparrow]

so not good, I hope it's just a temporary hiccup

[gusty sparrow]

yep it does this for large chunks of HTML for me too, not just recently - separate issue, but perhaps related

[final blaze]

Custom instructions

[modest spire]

Not sure if you're implying the custom instructions are causing issues. I did my testing without custom instructions. So that is not the problem for me.

[blissful belfry]

It seems to me that it's as if the context length is less than a full message length. I have an example here where ChatGpt4 is unable to see past about halfway through a given code snippet, with the total code snippet being around 350 lines long.

https://chat.openai.com/share/03a669c8-ca14-4d89-9e1a-ced412286cad

[gusty sparrow]

thanks @final blaze I thought this too, initiailly, and even though I turned custom instructions off and deleted them altogether, does this still affect it?

The problem has also persisted in other chat windows, old ones which have been running for months without custom instructions

[gusty sparrow]

Same here @modest spire I had since removed the custom instructions and the problem is universal across all conversations, including old ones 'pre-instructions era'

[fringe rain]

The API tells me that your initial post is 4000 tokens long, which I suspect is hitting the max length of chatGPT. When I use the GPT4 API (with an 8k context length) I get:
The following function implementations are visible in the provided code:

1. bool activated()
2. void init(i32 numRows, i32 numCols, i32 cellSize, sf::Vector2f spawnPosition)
3. void randomizeWeights(f32 min, f32 max)
4. void addEdgeVariability(f32 minRandomization, f32 maxRandomization)
5. void applyGaussianFilter()
6. void generateCellStates()
7. void draw(sf::VertexArray &vertexArray, bool drawEdges, bool drawPoints, bool drawTextWeights, bool drawTriangles)
8. sf::Vector2f interpolate(i32 index1, i32 index2)

[blissful belfry]

I'm still not entirely convinced. ChatGpt should have 4096 tokens, right? If it's 4,000 tokens for the entire thing, why is it stopping like 70% of the way through?

[fringe rain]

Because even my short response was 1000 tokens long

[blissful belfry]

Either way, I remember being able to post like 900 lines of code in the past, and I don't think this code is more dense than the code I posted in the past

[fringe rain]

Gpt3.5 has a higher context limit so it's possible you were using it.

[blissful belfry]

i wasn't using 3.5, but it doesn't even matter. Is there some technical limitation that prevents them from accurately analyzing the amount of tokens a message contains client side?

this situation is like an ignorance is bliss type of thing, because they allow you to feed in more than chat gpt can actually handle in a single message if what you're saying is true

[modest spire]

I'm not at all convinced here.

[modest spire]

Def not the case.

This is a clearly new problem with multiple users reporting the same.

[north bison]

Same here, it's been going in and out on its 'capabilities'. (This was on code interpreter)

[blissful belfry]

I'm really not sure anymore to be honest. I'm coming around to thinking that Kost Joist might be right, at least for my case

[modest spire]

I've been working with it in the same way for five months now. Doubt we'd all be experiencing the same symptoms randomly.

I have an app and am used to being able to load all four of the main files, and now, I chant even load one. Not making it up lol...

[blissful belfry]

a telling test would be finding an old conversation where you paste a large chunk of code, and repeating it today to see if it can't see the whole thing

[fringe rain]

This isn't a matter of opinion, anyone can take that text and paste it into the API to see the context length of the code is over 4k. And with the response it's over 5k.

[modest spire]

I went back to a conversation from about 3 weeks ago and repeated it. After passing the files, GPT claims to have never received them. To be clear, this is a repeat of a conversation that previously worked. I'm not sure what more I can say.Unfortunately, these are client projects that I can't really share here.

So, if something has changed about the function of the model and we can no longer work with code of any decent length, I'd love to know what. Have confirmed that this is not the case.

[modest spire]

Update: I tried the same conversation on a colleague's account.

I ensured the settings are the same. GPT 4 with no plugins enabled and no custom instructions.

Here's a summary of the conversation:

Prompt 1: I'm working on a web app. I have five files to send. I will send them one at a time.
Prompt 2: File 1/5 page.html:
Prompt 3: File 2/5 app.js:
Prompt 4: File 3/5 config.json:
Prompt 5: File 4/5 modifiers.json:
Prompt 6: File 5/5 style.css:
Prompt 7: Please summarize the function of the web app.
Prompt 8: Return page.html

On my account:

On my colleague's account:

I'm beginning to wonder if having had custom instructions enabled at some point in the past caused a bug that has persisted. This colleague has never had custom instructions enabled.
To anyone experiencing similar issues, have you all had custom instructions at some point?

[modest spire]

@gusty sparrow @blissful belfry I seem to have fixed this issue and was able to recreate it. It seems linked entirely to custom instructions. Once it was fixed, I was able to reproduce the issue by renabling custom instructions. Even having the switch for custom instructions on with no text in the input fields will cause this issue. To fix:

1. Enable custom instructions with some text in both input fields.
2. Run one prompt.
3. Delete text from both fields in custom instructions.
4. Toggle "Enable for new chats" to off.
5. Run one prompt.

To be clear, the following configuration will cause an error. (As will having text in either input)

I'll start a new bug report with the proper info now that we have a suspected cause.

[blissful belfry]

i just tried following the steps you outlined and couldn't replicate it 😦

[modest spire]

That sucks. I guess getting a new account would solve it, but that's a big pain.

[blissful belfry]

it's possible i did something wrong in the attempt or the length of each custom instruction field matters.

[gusty sparrow]

hey @modest spire can confirm the memory is working again. I didn't use ChatGPT for about a week and figured it was some bug relating to custom instuctions.

I still have not put custom instructions back in because I fear it will make ChatGPT unusable again. We are spending 4-5 hours using it each day right now as a critical tool in our workflow so I don't want to risk mucking that up again.

I guess custom instructions isn't truly reliable yet!

Thank you so much for looking into it.

[modest spire]

Glad it worked out! I was able to replicate the problem just by adding custom instructions, so yeah, avoiding then for now is a good bet.

[gusty sparrow]

hey @modest spire everyone else in this thread. GPT-4 chat it was working really well for about a week and now, as of today. The problem is back. Confirming I am not using any custom instructions.

Is it happening for anyone else?

[modest spire]

Let me take a look tomorrow and see how mine is doing; I haven't noticed anything yet.

[gilded fractal]

YUP!!!! this is one of its BIGGEST FAILURES and the devs jsut dont care... 😡

[gilded fractal]

YUP

[ripe zenith]

It seems like they're making chat lazier than it has been previously. It can't even pay attention to simple tasks anymore. And it constantly asks me redundantly if I'd like to proceed with the stuff I already told it to do. It's more like talking to a secretary answering 100 phone lines at once and can't juggle it all. I've even gotten other peoples responses.

[ripe zenith]

I've noticed that is sometimes works better than others. I suspect that at times of high traffic they have it deprioritize certain types of conversations (maybe those that have been going for more than a few minutes) so that they're using a dumbed down less capable version. I'm wondering if they're actually giving us chat 3.5 during times of high traffic...

[gusty sparrow]

Same here, we were writing javascript front-end conditional visibility code with it today and twice we received mathematica code for date & time-based formulas. Not a great experience as a paid user, even if we are only the consumer-facing peasants