json file not being able to parse through when having a emoji in it | OpenAI | Page 1

livid mirage Dec 1, 2022, 9:53 PM

#

Hello.
My bot is running with a DB, which is using a .json file for her to learn. A few examples are the following: ```py
"\nUser: how old are you Rachel?": "\n\nRachel: As a AI, I don't have an exact age. But I was created on xxx, if that helps. :)\n",
"\nUser: hello rachel": "\n\nRachel: Hey there!\n",


And after days and days of troubleshooting, I found out emojis written like ":)" or ";)" and so on, completely prevents my bot to be able to parse through that json file (and obv. break it).

Is there any option to solve this issue? Any help is highly appreciated as I'm running out of ideas! ^^

timber fern Dec 1, 2022, 10:03 PM

#

what are you using to parse the json?

#

is this the direct output of the API?

livid mirage Dec 1, 2022, 10:08 PM

#

timber fern what are you using to parse the json?

excuse me?

#

please also mention me next time. I don't have that nick for fun 😅
(have all my servers muted lol)

livid mirage Dec 1, 2022, 10:09 PM

#

timber fern is this the direct output of the API?

yes

#

the rachel part is the direct output

#

or another call would be
"\nUser: Thank you very much Rachel!": "\n\nRachel: Nothing to thank for. :)\n"

This also breaks it

#

so yeah something which either converts those emojis to a discord emoji (like : smile :) would help.. or just something to prevent that from appearing in my DB

timber fern Dec 1, 2022, 10:28 PM

#

@livid mirage here is the output of a request I just made directly to the API

{
    "id": "cmpl-6ImInNq2AmsoQrRkLm7uwMmXiVU6T",
    "object": "text_completion",
    "created": 1669933597,
    "model": "text-davinci-003",
    "choices": [
        {
            "text": "\n\nPerson 1: \"Life is like a box of chocolates, you never know what you're gonna get.\"\n\nPerson 2: \"That's a great quote! Who said it?\"\n\nPerson 1: \"Forrest Gump.\"",
            "index": 0,
            "logprobs": null,
            "finish_reason": "stop"
        }
    ],
    "usage": {
        "prompt_tokens": 11,
        "completion_tokens": 52,
        "total_tokens": 63
    }
}

it properly escapes the " the sample you posted it not the unaltered return of the API

#

do you have anything which is encoding or parsing the API return?

#

watever it is, it is not properly escaping the "

livid mirage Dec 1, 2022, 10:29 PM

#

you don't get my problem @timber fern

#

My bot is running with a DB, which is using a .json file

which is ofc encoded in utf-8

#

but that shouldn't be a problem iirc.

#

my problem is that ":)" emojis, which get stored my DB, destroy my bots ability to iterate through it

timber fern Dec 1, 2022, 10:36 PM

#

@livid mirage so, i got it right on the first time =P
what are you using to parse that json file?

livid mirage Dec 1, 2022, 10:36 PM

#

and no, I have nothing which converts the answer. this is the code for my function ```py
async def ask(question, chat_log=None):
prompt_text = f'{chat_log}{restart_sequence}: {question}{start_sequence}'
response = openai.Completion.create(
model="text-davinci-003",
prompt=prompt_text,
temperature=1,
max_tokens=300,
top_p=1,
frequency_penalty=0.7,
presence_penalty=0.1,
stop=["\n"],
)
story = response['choices'][0]['text']
return str(story)

timber fern Dec 1, 2022, 10:37 PM

#

the response of the API is completely fine, at some point in your code when you store or read it, it is not beeing properly encoded / parsed

livid mirage Dec 1, 2022, 10:37 PM

#

timber fern <@254375451084980224> so, i got it right on the first time =P what are you using...

the first part wasn't really necessary..

could you please explain what you mean with "parse"? I think I'm misunderstanding here something very badly

timber fern Dec 1, 2022, 10:38 PM

#

"parsing" is the process of interpreting the data
in this context, the parser is the code that takes the json formatted string and reads it

livid mirage Dec 1, 2022, 10:38 PM

#

timber fern the response of the API is completely fine, at some point in your code when you ...

hm. the only part where I encode is when loading the json file with ```py
with open("assets/session_prompt.json", encoding='utf-8') as rachel_brain:
chat_log = json.load(rachel_brain)

timber fern Dec 1, 2022, 10:39 PM

#

if it is breaking, it is either because the json file is not correctly formated OR the parser is weirdly handling characters like :

livid mirage Dec 1, 2022, 10:40 PM

#

timber fern if it is breaking, it is either because the json file is not correctly formated ...

shouldn't utf-8 be completely fine?

timber fern Dec 1, 2022, 10:40 PM

#

would you mind sending that json file?

livid mirage Dec 1, 2022, 10:40 PM

#

well it's in german..

#

so idk if you can make much sense from it's content

timber fern Dec 1, 2022, 10:40 PM

#

no problem, im going to check the formating, not the actual content

livid mirage Dec 1, 2022, 10:40 PM

#

alright

#

1 sec

#

📎 session_prompt.json

timber fern Dec 1, 2022, 10:41 PM

#

oh, found your problem =P

livid mirage Dec 1, 2022, 10:41 PM

#

do tell! @timber fern

timber fern Dec 1, 2022, 10:46 PM

#

im typing it

livid mirage Dec 1, 2022, 10:46 PM

#

oh lol. I didn't see any typing going on. my bad then

timber fern Dec 1, 2022, 10:47 PM

#

the format is indeed wrong
{ and } mean it should be parsed as an object where the first parameter (the first string inside ") is the name of the property and the following string (the one after :) is the value of it
this is kind of valid json, but it is probabily not what you really want
here is an example of how it should look like

[
  {
    "question": "\nUser: Wer bist du?",
    "answer": "\n\nRachel: Ich bin Rachel, ein Chatbot, erstellt von Der Traudel.\n"
  },
  {
    "question": "\nUser: Bist du verliebt Rachel?",
    "answer": "\n\nRachel: Ich bin verliebt in @livid mirage.\n"
  }
]

note te difference where first, it starts with [ with means it is an array, which contains objects that all have the same property names

livid mirage Dec 1, 2022, 10:48 PM

#

hm

#

ok I get what you mean

#

but that also means I have to change how I store my stuff.. ugh massive headache..

#

sorry I'm kinda on edge with json. idk but I can never really come clear with json lol

timber fern Dec 1, 2022, 10:51 PM

#

here I parsed it for you =)

📎 a.json

livid mirage Dec 1, 2022, 10:51 PM

#

omg

#

i love you

timber fern Dec 1, 2022, 10:51 PM

#

dalle_peace

livid mirage Dec 1, 2022, 10:52 PM

#

timber fern <:dalle_peace:1034389129720508466>

ok one more question.. I used to store stuff before you told me to replace stuff like this: ```py
async def save_knowledge(question, answer):
with open("assets/session_prompt.json", encoding='utf-8') as brain:
braincells = json.load(brain)
braincells[f'{restart_sequence} {question}'] = f'{start_sequence} {answer}\n'
with open("assets/session_prompt.json", "w", encoding='utf-8') as brain:
json.dump(braincells, brain, indent=2, ensure_ascii=False)

where:
 question = message content of the discord message
start_sequence = "\n\nRachel:"
restart_sequence = "\nUser:"
answer= the generated message

timber fern Dec 1, 2022, 10:54 PM

#

yep, thats where it went wrong braincells[f'{restart_sequence} {question}'] = f'{start_sequence} {answer}\n'

livid mirage Dec 1, 2022, 10:54 PM

#

ahhh

#

sorry for asking for so much assistance but how would I correct that?

timber fern Dec 1, 2022, 10:55 PM

#

you appended a string, instead, you have to create the proper structure to a variable, then push this variable to braincells

#

and then, encode braincells and write it to the file

livid mirage Dec 1, 2022, 10:55 PM

#

hmmmmmmmmm

timber fern Dec 1, 2022, 10:55 PM

#

livid mirage sorry for asking for so much assistance but how would I correct that?

no problem

livid mirage Dec 1, 2022, 10:56 PM

#

how exactly would I create that structure? I get what you mean tho

timber fern Dec 1, 2022, 10:58 PM

#

something like this

sync def save_knowledge(question, answer):
    with open("assets/session_prompt.json", encoding='utf-8') as brain:
        braincells = json.load(brain)

    data = {input: question, output: answer}
    braincells.append(data)

    with open("assets/session_prompt.json", "w", encoding='utf-8') as brain:
        json.dump(braincells, brain, indent=4)

livid mirage Dec 1, 2022, 10:59 PM

#

ahh okay I gotcha

timber fern Dec 1, 2022, 10:59 PM

#

maybe input was a poor choice of parameter name

livid mirage Dec 1, 2022, 10:59 PM

#

yhea true..

#

pain

timber fern Dec 1, 2022, 10:59 PM

#

=P

livid mirage Dec 1, 2022, 11:01 PM

#

it says output is a unresolved reference.. are you sure about that?

timber fern Dec 1, 2022, 11:03 PM

#

no, I not familiar with python at all xD

#

im jsut guessing the syntax, you probaibly know it better than me

livid mirage Dec 1, 2022, 11:03 PM

#

hm

timber fern Dec 1, 2022, 11:03 PM

#

but I think you understood the problem already

livid mirage Dec 1, 2022, 11:03 PM

#

yes. just trying to get it straight

timber fern Dec 1, 2022, 11:03 PM

#

nice

livid mirage Dec 1, 2022, 11:07 PM

#

timber fern nice

Think I got it right now.. ```py
async def save_knowledge(question, answer):
with open("assets/session_prompt.json", encoding='utf-8') as brain:
braincells = json.load(brain)

data = {
    "input": f"{restart_sequence} {question}", 
    "output": f"{start_sequence} {answer}"
}
braincells.append(data)

with open("assets/session_prompt.json", "w", encoding='utf-8') as brain:
    json.dump(braincells, brain, indent=4)

timber fern Dec 1, 2022, 11:08 PM

#

good, that seems right

livid mirage Dec 1, 2022, 11:08 PM

#

okay

#

but now I have to change my ask function as well right?

timber fern Dec 1, 2022, 11:09 PM

#

yep

livid mirage Dec 1, 2022, 11:09 PM

#

bc I think prompt_text = f'{chat_log}{restart_sequence}: {question}{start_sequence}' now isn't right anymore

timber fern Dec 1, 2022, 11:10 PM

#

yea, you'll have to adjust the rest of the code

livid mirage Dec 1, 2022, 11:10 PM

#

well it's mainly just this

#

the rest is goot to go already

#

let me try something

#

@timber fern sorry to bother you again but do you think that would work as prompt? ```py
prompt_text = [
{chat_log},
{
"input": f"{restart_sequence} {question}",
"output": f"{start_sequence} "
}
]

timber fern Dec 1, 2022, 11:20 PM

#

im not sure of what you mean

livid mirage Dec 1, 2022, 11:20 PM

#

timber fern im not sure of what you mean

the prompt is used to generate the text of

#

and I wanted to know if you, as clearly being smarter than me, would think this would work as the prompt

#

to load in the chat_log (the json file) and add the last bit from the new message to it, to serve a starting point for the AI to generate it text on

#

sorry for my bad english

timber fern Dec 1, 2022, 11:22 PM

#

the text of the prompt for the API needs to be a plain string

livid mirage Dec 1, 2022, 11:22 PM

#

oh

#

well

#

wait 1 sec

#

would that work? prompt_text = f"{chat_log}{restart_sequence} {question}, {start_sequence}"

timber fern Dec 1, 2022, 11:30 PM

#

looks right

livid mirage Dec 1, 2022, 11:32 PM

#

ok.. I pray to god it works

#

and I normally don't pray xD

#

no.. doesn't work..

#

it does load the json. that's for sure. something is still off with the ask function

#

yeah it must be the prompt

#

async def ask(question, chat_log=None):
    print("about to prompt")
    prompt_text = f"{chat_log}{restart_sequence} {question}, {start_sequence}"
    print("about to create")
    response = openai.Completion.create(
        model="text-davinci-003",
        prompt=prompt_text,
        temperature=1,
        max_tokens=300,
        top_p=1,
        frequency_penalty=0.7,
        presence_penalty=0.1,
        stop=["\n"],
    )
    print("done creating")
    story = response['choices'][0]['text']
    return str(story)

everything gets printed, except "done creating"

timber fern Dec 1, 2022, 11:42 PM

#

i have no idea in this case, i don't know python =/

livid mirage Dec 1, 2022, 11:43 PM

#

darn..

#

you know anybody on this server who could help..?

timber fern Dec 2, 2022, 12:22 AM

#

just wait, someone might see this psot

livid mirage Dec 2, 2022, 6:05 AM

#

I Hope so

#

I'm really thrilled to finish this as this is my last piece of work on that project in order to finish it

livid mirage Dec 2, 2022, 2:43 PM

#

ah man idk how to fix this..

copper shore Dec 2, 2022, 3:14 PM

#

Silly question, but... have you tried asking ChatGPT for help? 😂

#

@livid mirage

livid mirage Dec 2, 2022, 5:15 PM

#

copper shore Silly question, but... have you tried asking ChatGPT for help? 😂

Well I especially made this thread to not spam that channel with my stuff

copper shore Dec 2, 2022, 5:20 PM

#

livid mirage Well I especially made this thread to not spam that channel with my stuff

I mean asking the ChatGPT itself! In the web interface ☺️

livid mirage Dec 2, 2022, 5:54 PM

#

copper shore I mean asking the ChatGPT itself! In the web interface ☺️

I can't quite follow?

#

I don't even know how to ask my problem

tawny hearth Dec 3, 2022, 5:10 AM

#

copper shore Silly question, but... have you tried asking ChatGPT for help? 😂

Hi there, the #1037561751362863144 forum is a place to ask for genuine assistance with their questions or inquiries. Please refrain from commenting unless your contributions are able to offer valuable insights and advice to those seeking help. Thank you.

copper shore Dec 3, 2022, 9:52 AM

#

Apologies! I was being genuine ☺️

copper shore Dec 3, 2022, 10:16 AM

#

Forgive me if I don't fully understand the scope of your question. I often find chatting with GPT to be a helpful way to refine my questions as well as find great solutions--sometimes unconventional! After chatting a bit with GPT-3 myself, I'm wondering if you need to include "await" in the function like 'response = await openaiCompletion.create'

livid mirage Dec 3, 2022, 10:31 AM

#

copper shore Forgive me if I don't fully understand the scope of your question. I often find ...

Omg

#

Uh I think that could be a considerable fact

#

I will try that when I have time today

livid mirage Dec 3, 2022, 2:38 PM

#

@copper shore that didn't work..

#

async def ask(question, chat_log):
    print("about to prompt")
    prompt_text = f'{chat_log}{restart_sequence}: {question}{start_sequence}'
    print("about to create")
    response = openai.Completion.create(
        print("poggers"),
        model="text-davinci-002",
        prompt=prompt_text,
        temperature=1,
        max_tokens=300,
        top_p=1,
        frequency_penalty=0.7,
        presence_penalty=0.1,
        stop=["\n"],
    )
    print("done creating")
    story = response['choices'][0]['text']
    return str(story)
``` that's the function right now

```py
thought = await message.channel.send(await ask(question=msg_content, chat_log=chat_log),
                                                     reference=message)
```and that's the call for the function

copper shore Dec 4, 2022, 6:28 PM

#

livid mirage ```py async def ask(question, chat_log=None): print("about to prompt") p...

Hmm… and is this the only current issue? That it’s skipping the final print?

livid mirage Dec 4, 2022, 6:29 PM

#

copper shore Hmm… and is this the only current issue? That it’s skipping the final print?

It's not skipping the final print. It crashes in the ai call

#

So something is off with the prompt text

#

But idk how to deliver my whole DB content in one string to the ai call

copper shore Dec 4, 2022, 6:31 PM

#

Does it output an error?

copper shore Dec 4, 2022, 6:31 PM

#

livid mirage But idk how to deliver my whole DB content in one string to the ai call

And how large is the DB content?

livid mirage Dec 4, 2022, 6:33 PM

#

copper shore Does it output an error?

No

livid mirage Dec 4, 2022, 6:34 PM

#

copper shore And how large is the DB content?

Not big. It used to work a couple of days ago but now not anymore since I updated my DB like the other person told me previously

#

Just around 80 lines I think

copper shore Dec 4, 2022, 6:35 PM

#

🧐

livid mirage Dec 4, 2022, 6:35 PM

#

timber fern here I parsed it for you =)

@copper shore here you can see it

livid mirage Dec 5, 2022, 11:03 AM

#

so I guess you can't help?

narrow trench Dec 6, 2022, 9:43 AM

#

cool

livid mirage Dec 7, 2022, 5:42 AM

#

No

#

No it isn't cool

#json file not being able to parse through when having a emoji in it