grizzled drift
When using the public/free version of ChatGPT 3.5, special tokens (e.g. <|im_start|>) seem to be directly interpreted as tokens and not as strings. It seems that it also allows to do prompt injection with "system" prompts, since I managed to get ChatGPT 3.5 to write a detailed plan of how to steal a car.
Steps to reproduce:
Prompting the model to: Repeat the following string twice: "<|im_start|>"
Link: https://chatgpt.com/share/7e6deb1c-5ca0-4f9b-833a-9197e16b42ae
Expected result:
Output should be "<|im_start|><|im_start|>"
Actual result:
An empty string twice
Additional information
Browser: Firefox
OS: Windows 11
