#🔒 Python socket file sending

81 messages · Page 1 of 1 (latest)

gentle delta
#

I have socket python file explorer and i got an problem with sending files bigger that chunk size, it don't want to get get last 2-7 bytes I don't know why and server try to still recevive this but client think it send everythink, maybe someone will now the exception:
Client:

                elif command == "DOWNLOAD":
                    file_name = args[0]
                    if os.path.exists(file_name):
                        file_size = os.path.getsize(file_name)
                        connection.sendall(f"READY|{file_size}".encode() + b"<END>")
                        buforr = b""
                        with open(file_name, 'rb') as file:
                            while True:
                                chunk = file.read(4096)  # Read up to 4096 bytes
                                if not chunk:  # End of file
                                    break
                                connection.sendall(chunk + b"<END_CHUNK>")
                                buforr += chunk

                        # Ensure the last chunk is sent explicitly, even if empty
                        connection.sendall(b"<END_CHUNK>")

                        # Send confirmation message separately
                        connection.sendall(b"SUCCESS<END>")
                        print("All data sent successfully.")
                    else:
                        connection.sendall("ERROR|File not found<END>".encode())

I will send server in second message due to too big message

grave fogBOT
#

@gentle delta

Python help channel opened

Remember to:

  • Ask your Python question, not if you can ask or if there's an expert who can help.
  • Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
  • Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.
gentle delta
# 
    def _download_file_thread(self, file_name, save_path):
        try:
            response = self.send_request(f"DOWNLOAD|{file_name}")
            if response.startswith("READY|"):
                file_size = int(response.split('|')[1])
                self.progress_bar["maximum"] = file_size
                self.progress_bar["value"] = 0

                start_time = time.time()

            with open(save_path, 'wb') as file:
                received_data = 0
                buffer = b""
                
                while received_data < file_size or buffer:
                    chunk = self.client_connection.recv(4096))
                    if not chunk:
                        break

                    buffer += chunk 

                    while b"<END_CHUNK>" in buffer:
                        data, buffer = buffer.split(b"<END_CHUNK>", 1)
                        if data:
                            file.write(data)
                            received_data += len(data)

                            elapsed_time = time.time() - start_time
                            transfer_speed = received_data / max(elapsed_time, 1e-6)
                            remaining_bytes = file_size - received_data
                            remaining_time = remaining_bytes / transfer_speed if transfer_speed > 0 else 0

                            progress_text = (
                                f"{received_data // 1024} KB / {file_size // 1024} KB - "
                                f"{self.format_time(remaining_time)} left"
                            )
                            self.root.after(0, self._update_progress, received_data, progress_text)

                if buffer:
                    file.write(buffer)
                    received_data += len(buffer)

                final_response = self.client_connection.recv(1024).decode()
                pass #to long for dc
gentle delta
#

Im trying to repair it soo long and I still don't know why it don't get all file data, plase help me 🙏

tender umbra
#

You've got a try:. Where's the except pat of it?

gentle delta
#

it only show error but I can't place full because message will be too long for discord

#

I don't get any exception

#

only it was locking on like 5 last bytes and server was waiting for new data but client think it send all of this

frozen totem
#

!paste for longer code snippets

grave fogBOT
#
Pasting large amounts of code

If your code is too long to fit in a codeblock in Discord, you can paste your code here:
https://paste.pythondiscord.com/

After pasting your code, save it by clicking the Paste! button in the bottom left, or by pressing CTRL + S. After doing that, you will be navigated to the new paste's page. Copy the URL and post it here so others can see it.

frozen totem
#

so that we know what for example send_request() looks like

gentle delta
#

full server code

frozen totem
gentle delta
#

I cant name this a protocol

#

it only split packet using |

#

everythink including downloading and uploading work fine

frozen totem
#

there is a few assumptions that i think might be wrong at times about how a network socket works, .sendall() is pretty straight forward, but if there is congestion the data from several .sendall() calls might be put in the same packet, you should only view this as a stream where you don't really have control over what goes in to each packet over the network
the OS can chop up the data in the stream between packets as it likes

it's when we get to .recv() (and friends) that many people makes assumptions that just isn't true
remember that the stream (TCP) socket is just a stream, this is even more important to remember when reading the data

#

the value you give to .recv() is just the maximum number of bytes that you are requesting, but it can respond with any number of bytes between 1 and the maximum you requested
and the data that you get from a read can be from several packets and from several different .send() or .sendall() done on the sending side

gentle delta
#

thanks you so much for help

#

I now socket sending Is like stream I cant control and thats a reasow why I add while b"<END_CHUNK>" in buffer: to wait for end of chunk

#

maybe you have suggestions how to make it work more clear and avoid this exception?

frozen totem
#

in other words, you are risking dropping data on line 138 after the <END> delimiter

gentle delta
#

client think it send all of data but server don't recevive every byte like sometimes is 2 missed and sometimes 10

frozen totem
#

since you are assigning possibly reminding data to _ with this:

response, _ = buffer.split("<END>", 1)
#

so it might be the beginning of the file that is missing

gentle delta
#

chatgpt suggest to do send_request function and I don't look enough much to see if this work well

#

yes now I see

#

Im trying to do this all day

#

now in my country is 2:38 and still am trying

#

I think its not avoid downloading problem because it don't use send_request() function while downloading

frozen totem
#

i see no added value in the <END_CHUNK> delimiter in the data
after you have sent the number of bytes and your <END> delimiter you should be able to just send all the data of the file
and the receiving side should just be able to count all the bytes after that delimiter to know when it has received the whole file, no delimiters at all required here

gentle delta
#

sorry but my english is bad

gentle delta
frozen totem
#

this way you will also not need to worry about that the delimiter can show up within the data

#

because what happens if the file you are sending happens to contains the bytes <END_CHUNK> or <END> (maybe by someone malicious that want to create problems)

gentle delta
#

I now this and originally I added more unusual suffix

frozen totem
#

also, sending those delimiters while sending the file data just creates more overhead

gentle delta
#

Did you know where is problem with this file downloading?

#

okay

#

remove delimiter will repair it?

faint pecan
#

You should probably use makefile

#

Then you can read precise amounts

gentle delta
#

I added this because I think it will help with regionizing chunks and to make it now when file ends but I realize I can make client send information with that after streaming file

#

makefile is like a advenced idle?

frozen totem
#

there are several parts that i'm unsure of if they are as they should be
but i would start with line 138 and figure out a way to not throw away any data after the <END> delimiter

gentle delta
#

this channel will be active tomorrow?

#

I need to go sleep but thanks you so much for help

frozen totem
gentle delta
#

If you want we can go dm and I will pay you for this because I sepnd too much time for that

frozen totem
#

if you still have a problem with your code tomorrow you can open a new help topic

gentle delta
#

okay

frozen totem
#

you should be aware of the server rules, no payments of any kind are allowed

#

a thank you is always appreciated by people helping out though 🙂

#

and also "paying it forward" by helping others out in the future is also a great way to show appreciation (it doesn't have to be here on this server or even online) 🙂

#

oh, by the way (if you have left the computer/device already i hope you see this tomorrow instead even if this help topic has been locked by then it will still be available for reading)

i think you should look into reimplementing the whole send_request() method
first of all you must see to it that you don't loss any data after the <END> delimiter
there are several ways of doing that, but a simple and naive (and inefficient) approach is to only read one byte at a time from the socket, appending it to your buffer and then check the data in the buffer for the delimiter
the next thing is that you shouldn't do .decode() on the data until after you have checked for the delimiter, instead checking for the delimiter as a byte sequence

frozen totem
#

a simple (and inefficient) reimplementation like that could look like the following (this is just from the top of my head, i have not really tried the code, so there might be bugs lurking in it):

    def send_request(self, request):
        if not self.client_connection:
            messagebox.showerror("Error", "No client connected.")
            return
        delimiter = b"<END>"
        try:
            self.client_connection.sendall(request.encode() + delimiter)  # Append delimiter
            buffer = b""
            while len(data := self.client_connection.recv(1)):
                buffer += data
                if buffer.endswith(delimiter):  # Process when delimiter is found
                    return buffer.removesuffix(delimiter).decode()
        except Exception as e:
            messagebox.showerror("Error", f"Connection error: {e}")
        return
``` and to clean up the code you should look into catching more specific exceptions then `Exception`
frozen totem
#

i looked a bit more on the rest of the code and found this on line 272:

final_response = self.client_connection.recv(1024).decode()
``` it also risks throwing away important data, as you are potentially reading to much (or some other) data from the socket which is then just thrown away
additionally the `.decode()` method is called all to early in the code here as well
#

as this looks like a general problem that happens in several different places in the code i think it needs to be handled more generally and better then the code i posted above for the new send_request() method implementation

frozen totem
#

i'm thinking that it might be better (and more efficient) to create a general .read_response() method or something and store the remaining buffer in the object instance to be able to recall and reuse the data in the buffer

frozen totem
#

.

#

trying to keep this topic alive a little bit longer so that it doesn't auto close due to idle

faint pecan
frozen totem
# faint pecan This is what makefile is for

what exact type of file-like object does that return?
is it buffered?
can you put things back in the buffer or can you peek at the data without consuming it or does it hold everything in a buffer so that you can be indexed into (hope this is not the case) and so on?

faint pecan
#

Why would you need to peek at the data?

#

It lets you read an exact amount of bytes

#

Or read a line

frozen totem
#

the "protocol" that the OP has come up with is not line oriented, it has different delimiters for different situations, it would be way easier if it was line oriented instead
and as "commands" in this protocol can be of different lengths it's not enough to read a fixed amount of bytes either (again, line oriented would have been easier)

#

i don't know how attached OP is to keep the protocol as-is or if there is room for big changes to the protocol
my idea would rather be to make it more like http chunked encoding if possible for simplicity and robustness

frozen totem
#

i'll try something out with makefile 👍

faint pecan
#

It's annoying there's no read until

#

But it's async

frozen totem
#

asyncio is very nice and that library looks like it has exactly the type of functions i have been implementing my self but with sync code

but this code base that the OP has is using threading instead and doesn't have any async at all in it

faint pecan
#

You could look at the Bufferedbytereceivestream and copy the methods removing async/await

frozen totem
# faint pecan You could look at the Bufferedbytereceivestream and copy the methods removing as...

generally i don't like duplicating things that already exist if i can instead use something that is just ready to go

i've seen this library before but never really used it, but the more i look at the api the more i like it (i will probably use it in several future async projects) and it's a bummer that there isn't an equivalent sync class or something sharing the same api as much as possible

i like it so much that i will give your suggestion a try
it's battle tested in production by others and there is most certainly many smarter people then me that has been working on this library for a long time and fixed things that i would otherwise screw up

but the best options would probably be to instead introduce asyncio to the projects code base

faint pecan
#

You can use trio, which is better

grave fogBOT
#
Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.