🔒 For some reason this script keeps creating duplicates inside of the json file. | Python | Page 1

night tendonBOT Jul 23, 2024, 2:38 AM

#

@waxen sail

Python help channel opened

Remember to:

Ask your Python question, not if you can ask or if there's an expert who can help.
Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

waxen sail Jul 23, 2024, 2:39 AM

#

https://paste.pythondiscord.com/M2WA

#

{
    "560607653097766913": [
        {
            "role": "user",
            "content": "@abstract wave Hello!"
        },
        {
            "role": "assistant",
            "content": "Hello! How can I assist you today?"
        },
        {
            "role": "user",
            "content": "Got anything to show me?"
        },
        {
            "role": "assistant",
            "content": "I'm sorry, but I currently don't have the capability to provide visual or interactive content like images, videos, or text-based information. However, I can offer support in areas like education, technology, science, and general knowledge. Please let me know if there's anything specific you'd like help with!"
        }
    ],
    "560607653097766913": [
        {
            "role": "user",
            "content": "Hello!"
        },
        {
            "role": "assistant",
            "content": "Hello! How can I assist you today?"
        }
    ]
}

#

# Initialize conversation history for the user if not already present
    if user_id not in conversation_histories:
        conversation_histories[user_id] = []

I did this, so I have 0 clue why it's still creating duplicates

pseudo flare Jul 23, 2024, 2:58 AM

#

that seems unlikely given that you're using a dict

waxen sail Jul 23, 2024, 3:00 AM

#

pseudo flare that seems unlikely given that you're using a dict

?

#

Whats unlikely?

pseudo flare Jul 23, 2024, 3:01 AM

#

a dict can't hold duplicate keys

waxen sail Jul 23, 2024, 3:01 AM

#

Your right, which is why it's erroring

#

It isn't supposed to have duplicates

pseudo flare Jul 23, 2024, 3:01 AM

#

it.. doesn't. it can't.

waxen sail Jul 23, 2024, 3:01 AM

#

It's doing it right now

#

I mean I can get a video lol

waxen sail Jul 23, 2024, 3:05 AM

#

pseudo flare it.. doesn't. it can't.

#

You where saying?

frigid rover Jul 23, 2024, 3:21 AM

#

@waxen sail it might be easier if you have the json as a defaultdict when it's loaded into the program

#

!defaultdict

night tendonBOT Jul 23, 2024, 3:21 AM

#

The `collections.defaultdict` class

The Python defaultdict type behaves almost exactly like a regular Python dictionary, but if you try to access or modify a missing key, the defaultdict will automatically insert the key and generate a default value for it.
While instantiating a defaultdict, we pass in a function that tells it how to create a default value for missing keys.

>>> from collections import defaultdict
>>> my_dict = defaultdict(int)
>>> my_dict
defaultdict(<class 'int'>, {})

In this example, we've used the int class which returns 0 when called like a function, so any missing key will get a default value of 0. You can also get an empty list by default with list or an empty string with str.

>>> my_dict["foo"]
0
>>> my_dict["bar"] += 5
>>> my_dict
defaultdict(<class 'int'>, {'foo': 0, 'bar': 5})

Check out the docs to learn even more!

frigid rover Jul 23, 2024, 3:23 AM

#

import json
from collections import defaultdict

with open('./data.json') as f:
    data = defaultdict(list, json.load(f))

waxen sail Jul 23, 2024, 3:30 AM

#

frigid rover ```py import json from collections import defaultdict with open('./data.json') ...

# Load conversation history from file if it exists
if os.path.exists(history_file):
    print(f"{history_file} found! Loading conversation history from file.")
    try:
        with open(history_file, 'r') as file:
            data = defaultdict(list, json.load(file))
    except json.JSONDecodeError:
        print("Error loading JSON file. File may be corrupted.")

#

Is this correct?

#

I'm a little confused on what im needing to change and why is all

pseudo flare Jul 23, 2024, 3:32 AM

#

making it reproducable for others would let them figure it out in 30 seconds

waxen sail Jul 23, 2024, 3:32 AM

#

Well I gave the code

pseudo flare Jul 23, 2024, 3:32 AM

#

technically we could go get api keys, or have them

waxen sail Jul 23, 2024, 3:33 AM

#

But I dont think anyone wants to really go through that trouble ya know?

austere cedar Jul 23, 2024, 3:33 AM

#

waxen sail You where saying?

They're not wrong; you're doing some weird things with your dictionary such as using append for a list that exists inside the dictionary.

conversation_histories[user_id].append({"role": "user", "content": user_message})

So, yeah, it then makes sense to have duplicates.

waxen sail Jul 23, 2024, 3:34 AM

#

Ohh

pseudo flare Jul 23, 2024, 3:34 AM

#

that's..not weird

waxen sail Jul 23, 2024, 3:34 AM

#

What should I be doing then?

#

I was given multiple solution that I dont even know how to execute. Perhaps an example and explanation on what I need to do would be rather helpful so I understand what im doing.

pseudo flare Jul 23, 2024, 3:37 AM

#

I don't think you've been given any solution or explanation

#

I think you're doing something that you're not showing, which you may be unaware of

#

a dict cannot hold duplicate keys, meaning that is not what is happening meaning something else is happening

waxen sail Jul 23, 2024, 3:38 AM

#

I showed a video, showed the code

#

the only thing I didnt show was the video recording of me on the discord side communicating with the bot

austere cedar Jul 23, 2024, 3:38 AM

#

waxen sail What should I be doing then?

If you don't want duplicates, don't use a list instead use .update to add to the dictionary.

pseudo flare Jul 23, 2024, 3:38 AM

#

you're doing things in the background and you could for instance have multiple bots overwriting files

pseudo flare Jul 23, 2024, 3:38 AM

#

austere cedar If you don't want duplicates, don't use a list instead use `.update` to add to t...

a dict can't have duplicates, and there's nothing weird about appending to a list in a dict

waxen sail Jul 23, 2024, 3:39 AM

#

pseudo flare you're doing things in the background and you could for instance have multiple b...

I only have 1 instance open

#

I mean I can make the bot go offline when I kill the script

waxen sail Jul 23, 2024, 3:39 AM

#

austere cedar If you don't want duplicates, don't use a list instead use `.update` to add to t...

I'll try this

austere cedar Jul 23, 2024, 3:39 AM

#

pseudo flare a dict can't have duplicates, and there's nothing weird about appending to a lis...

a list inside a dict can have duplicates

#

here's an example

pseudo flare Jul 23, 2024, 3:40 AM

#

waxen sail I only have 1 instance open

something actually useful: you could print out the dict before you save it to file, you can also write out the file content before/after

#

def save_conversation_history():
    print("saving conv history:", conversation_histories)
    try:
        with open(history_file, 'w') as file:
            json.dump(conversation_histories, file, indent=4)
    except Exception as e:
        logging.error(f"Error while saving conversation history: {e}")

austere cedar Jul 23, 2024, 3:41 AM

#

there's also another issue with the code but i have to go
can someone cover the global keyword issue for the user? thanks

waxen sail Jul 23, 2024, 3:41 AM

#

austere cedar If you don't want duplicates, don't use a list instead use `.update` to add to t...

Traceback (most recent call last):
  File "C:\Users\Zayden\AppData\Local\Programs\Python\Python310\lib\site-packages\nextcord\client.py", line 497, in _run_event
    await coro(*args, **kwargs)
  File "c:\Users\Zayden\Downloads\Ollama API Call Test\ollama_bot.py", line 103, in on_message
    await handle_chat(message)
  File "c:\Users\Zayden\Downloads\Ollama API Call Test\ollama_bot.py", line 62, in handle_chat
    conversation_histories[user_id].update({"role": "user", "content": user_message})
AttributeError: 'list' object has no attribute 'update'

pseudo flare Jul 23, 2024, 3:41 AM

#

austere cedar there's also another issue with the code but i have to go can someone cover the...

seems like something to cover after addressing the bug

pseudo flare Jul 23, 2024, 3:42 AM

#

waxen sail ``` Traceback (most recent call last): File "C:\Users\Zayden\AppData\Local\Pro...

they meant to update the dict, but it's not going to help in any way, I have no clue what they're on about with that

#

your appending is totally valid, and, as mentioned, a dict can't have duplicate keys so it can't cause this issue

#

conceivably doing the file save wrong could cause weird results but I don't see it. like opening in rw, but that would cause a different result from what you're getting anyway

waxen sail Jul 23, 2024, 3:44 AM

#

pseudo flare conceivably doing the file save wrong could cause weird results but I don't see ...

PS C:\Users\Zayden\Downloads\Ollama API Call Test> & C:/Users/Zayden/AppData/Local/Programs/Python/Python310/python.exe "c:/Users/Zayden/Downloads/Ollama API Call Test/ollama_bot.py"
Logged in as OpenAI Bot#5927 (ID: 1265086266702037136)
Saving convo history: {560607653097766913: [{'role': 'user', 'content': 'Hello!'}, {'role': 'assistant', 'content': 'Hello! How can I assist you today?'}]}
Saving convo history: {560607653097766913: [{'role': 'user', 'content': 'Hello!'}, {'role': 'assistant', 'content': 'Hello! How can I assist you today?'}, {'role': 'user', 'content': 'This is a test conversatioN!'}, {'role': 'assistant', 'content': 'That sounds interesting. What topic would you like us to discuss today?'}]}
PS C:\Users\Zayden\Downloads\Ollama API Call Test> & C:/Users/Zayden/AppData/Local/Programs/Python/Python310/python.exe "c:/Users/Zayden/Downloads/Ollama API Call Test/ollama_bot.py"
conversation_histories.json found! Loading conversation history from file.
Logged in as OpenAI Bot#5927 (ID: 1265086266702037136)
Saving convo history: {'560607653097766913': [{'role': 'user', 'content': 'Hello!'}, {'role': 'assistant', 'content': 'Hello! How can I assist you today?'}, {'role': 'user', 'content': 'This is a test conversatioN!'}, {'role': 'assistant', 'content': 'That sounds interesting. What topic would you like us to discuss today?'}], 560607653097766913: [{'role': 'user', 'content': 'This is another test conversation'}, {'role': 'assistant', 'content': "Sure, I'm happy to assist with any further questions or conversations you have. Just let me know how I can help."}]}

pseudo flare Jul 23, 2024, 3:45 AM

#

well that's wild

#

oh but

#

this shows the issue

waxen sail Jul 23, 2024, 3:45 AM

#

?

pseudo flare Jul 23, 2024, 3:45 AM

#

problem solved.

#

pretty-print/format that dict and look at it

waxen sail Jul 23, 2024, 3:46 AM

#

How?

pseudo flare Jul 23, 2024, 3:46 AM

#

idk paste it in a .py file and run your formatter or do it manually

waxen sail Jul 23, 2024, 3:46 AM

#

Sorry I might be missing something here in the terminal output

pseudo flare Jul 23, 2024, 3:46 AM

#

yeah. and it's super obvious if you paste that dict you printed into an editor

waxen sail Jul 23, 2024, 3:47 AM

#

pseudo flare Jul 23, 2024, 3:47 AM

#

sec

#

{
    "560607653097766913": [
        {"role": "user", "content": "Hello!"},
        {
            "role": "assistant",
            "content": "Hello! How can I assist you today?",
        },
        {"role": "user", "content": "This is a test conversatioN!"},
        {
            "role": "assistant",
            "content": "That sounds interesting. What topic would you like us to discuss today?",
        },
    ],
    560607653097766913: [
        {"role": "user", "content": "This is another test conversation"},
        {
            "role": "assistant",
            "content": "Sure, I'm happy to assist with any further questions or conversations you have. Just let me know how I can help.",
        },
    ],
}

waxen sail Jul 23, 2024, 3:47 AM

#

Its saving it as a string

pseudo flare Jul 23, 2024, 3:48 AM

#

see it?

waxen sail Jul 23, 2024, 3:48 AM

#

no not really

pseudo flare Jul 23, 2024, 3:48 AM

#

the keys aren't duplicate in that

waxen sail Jul 23, 2024, 3:48 AM

#

waxen sail

Here it looks like a string tho

pseudo flare Jul 23, 2024, 3:49 AM

#

    "560607653097766913": [
    560607653097766913: [

waxen sail Jul 23, 2024, 3:49 AM

#

AH

#

Whats causing that

#

where's the code thats causing that

#

little bugger

pseudo flare Jul 23, 2024, 3:50 AM

#

Stelercus' suggestion would in fact fix that, maybe they spotted the problem already

waxen sail Jul 23, 2024, 3:50 AM

#

I didnt understand there solutuion tho

pseudo flare Jul 23, 2024, 3:50 AM

#

since then you'd only set in one place, currently you set in two places

#

and one of them sets as integer and the other as string

#

presumably, haven't read the code

waxen sail Jul 23, 2024, 3:50 AM

#

Well the problem looks like for some reason it's loading it as a string, when it was originally a int

pseudo flare Jul 23, 2024, 3:50 AM

#

ah

#

oh. you're right.

#

so. json can't have integer keys, keys are string, always

#

so you'll have the encoder turn the int into a string and then load it back

waxen sail Jul 23, 2024, 3:52 AM

#

So what do I do?

pseudo flare Jul 23, 2024, 3:52 AM

#

i suppose their suggestion might not fix it then

pseudo flare Jul 23, 2024, 3:53 AM

#

waxen sail So what do I do?

if you have int and string and need them to both be the same, then you'd change one or the other

waxen sail Jul 23, 2024, 3:54 AM

#

Well i'm confused how or why

#

one second

austere cedar Jul 23, 2024, 3:55 AM

#

waxen sail ``` Traceback (most recent call last): File "C:\Users\Zayden\AppData\Local\Pro...

append is a function that is only for lists; update is a function for dictionaries

pseudo flare Jul 23, 2024, 3:56 AM

#

(btw, json is a terrible database format re-writing the entire thing for any update is a lot of work. for trivial use though it works)

waxen sail Jul 23, 2024, 3:57 AM

#

so what do I do to force it to be a string?

#

because user_id = message.author.id gives me a number, which I prefer anyways.

austere cedar Jul 23, 2024, 3:57 AM

#

use str() to cast

waxen sail Jul 23, 2024, 3:59 AM

#

so conversation_histories[str(user_id)].append({"role": "user", "content": user_message})

austere cedar Jul 23, 2024, 3:59 AM

#

the casting is correct

waxen sail Jul 23, 2024, 4:01 AM

#

seems to want to override now\

austere cedar Jul 23, 2024, 4:02 AM

#

just want to ask some clarification questions: the duplicates we're talking about are the user id right?

waxen sail Jul 23, 2024, 4:02 AM

#

austere cedar just want to ask some clarification questions: the duplicates we're talking abou...

Correct

austere cedar Jul 23, 2024, 4:03 AM

#

why do you not want duplicates in this conversation?
is it because u want the nested json structure to continue the entire conversation for that specific user?

waxen sail Jul 23, 2024, 4:04 AM

#

waxen sail

I fixed this aswell

waxen sail Jul 23, 2024, 4:04 AM

#

austere cedar why do you not want duplicates in this conversation? is it because u want the ne...

The reason I dont is because the conversation is acting like a sort of

#

chat history for a chatgpt model

#

There can't be 2 users with the same user id

#

Thats just not possible

#

So each user ID gets its own chat history

austere cedar Jul 23, 2024, 4:05 AM

#

ah okay

waxen sail Jul 23, 2024, 4:05 AM

#

Now my problems seems to be the chat model doesn't want to remember stuff from the history

#

AAAA

#

  try:
        response = client.chat.completions.create(
            model="qwen2:1.5b",
            messages=[
                {"role": "system", "content": f"The user's display name is {display_name}"},
                *conversation_histories[str(user_id)]
            ]
        )
``` that has something to do with this

#

maybe this is why I need that global value?

#

global conversation_histories

waxen sail Jul 23, 2024, 4:07 AM

#

waxen sail ```py try: response = client.chat.completions.create( mode...

For context, if you want to make a basic request

#

from openai import OpenAI
client = OpenAI()

completion = client.chat.completions.create(
  model="gpt-4o-mini",
  messages=[
    {"role": "system", "content": "You are a poetic assistant, skilled in explaining complex programming concepts with creative flair."},
    {"role": "user", "content": "Compose a poem that explains the concept of recursion in programming."}
  ]
)

print(completion.choices[0].message)

austere cedar Jul 23, 2024, 4:08 AM

#

yeah i know; the llm is storing the conversation inside a list--its a common thing for it remember conversation history

waxen sail Jul 23, 2024, 4:08 AM

#

For some reason tho,

#

It's not reading the data correctly?

austere cedar Jul 23, 2024, 4:11 AM

#

There's a lot to cover: the issue why the help is spotty is because your code is the entire raw code.

It's not a minimal reproducible example: https://stackoverflow.com/help/minimal-reproducible-example, so it makes it difficult to actually try and help.

pseudo flare Jul 23, 2024, 4:16 AM

#

yeah it's difficult to figure things out when you don't know what you can trust, so not being able to reproduce something really messes the other person

austere cedar Jul 23, 2024, 4:16 AM

#

The reason you can't get out of using global is that there's a lack of understanding of how Python works--no one should be using local or global: it just messes with the scope of functions--its unnatural. We use if __name__ == __main__: main() excuse the one liner. The main() function in python is where you make calls to your functions and pass in that pesky global variable you got there. You can do more research on this.

pseudo flare Jul 23, 2024, 4:16 AM

#

because 75% of your energy goes to second guessing everything

austere cedar Jul 23, 2024, 4:17 AM

#

yeah ^

#

also its 12 am for me so its even worse lol

pseudo flare Jul 23, 2024, 4:17 AM

#

time zones are a thing

#

the global should be fine unless one starts using some fancy way to execute the code such as reloading the module

#

stinky or not, if it works it works

night tendonBOT Jul 23, 2024, 5:20 AM

#

Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.

#🔒 For some reason this script keeps creating duplicates inside of the json file.