#πŸ”’ Stream dynamic zipfile.ZipFile to urllib3.request() or requests.post()

45 messages Β· Page 1 of 1 (latest)

full prism
#

The zip file can't fit in a memory buffer (e.g. io.BytesIO) and it won't fit in disk as well. So it must be sent as it is generated

The output of zipfile.ZipFile is a .write() -able fileobj.
The input of urllib3.request(): body parameter (i guess an iterable or .read() -able object are the only options here)

  • The input of requests.post(): data parameter (same as urllib3.request)

The problem is that I can't seem to connect the output of ZipFile and the input of urllib3.request() (body parameter) or requests.post() (data parameter)

urban galeBOT
#

@full prism

Python help channel opened

Remember to:

  • Ask your Python question, not if you can ask or if there's an expert who can help.
  • Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
  • Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.
still matrix
full prism
still matrix
jaunty carbon
#

The issue is
How do you live generate the zip file

still matrix
#

where are you reading the zipfile from? since you say it won't fit on disk or memory

full prism
jaunty carbon
#

It still would have to process the entire thing before writing it, which means it would have store it in the memory

full prism
violet prairie
#

writestr? So you have the full bytes to send in memory?

full prism
still matrix
violet prairie
#

Or you have multiple calls to writestr?

full prism
still matrix
#

can you show your code?

full prism
violet prairie
#

Use a BytesIO that you truncate between writes

still matrix
versed spruce
#

any Good coder

#

python coder

#

i need bit information

still matrix
#

?

#

this help channel is taken, please see #β“ο½œhow-to-get-help

versed spruce
#

i need just information about code not even code now

#

is that possible to make a python code which will auto fill captcha

still matrix
versed spruce
#

of any kind

full prism
# still matrix can you show your code?

I prefer not to show exactly what is generated.

But to give an idea. An equivalent code would generate the zip dynamically like this

for i in range(10000):
  zip.writestr(f"file{i}", base64.b64encode(os.urandom(1024)))
versed spruce
#

alr

jaunty carbon
still matrix
full prism
still matrix
#

should look like
zip = zipfile.ZipFile(...)
or

with zipfile.ZipFile(...) as zip:
  ...
#

i.e how is zip defined

full prism
# 
with zipfile.ZipFile(file=output, mode="w") as zip:
  for i in range(10000):
    zip.writestr(f"file{i}", base64.b64encode(os.urandom(1024)))

then output is a .write() -able fileobj

violet prairie
# 
def gen_zip(gen):
    buff = io.BytesIO()
    with zipfile.ZipFile(buff, 'w') as zf:
        for filename, data in gen:
            zf.writestr(filename, data)
            yield buff.getvalue()
            buff.truncate(0)
    yield buff.getvalue()
    ```
still matrix
#

is the zipfile library ok with working with an incomplete buffer?

full prism
still matrix
violet prairie
#

it only seeks if you use 'r' or 'a'

still matrix
#

so yeah i think i agree with what graingert suggested there by periodically flushing it out, so to speak

full prism
urban galeBOT
#
Python help channel closed for inactivity

This help channel has been closed. Feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.