🔒 Need help with AI solution where we interface with OpenAI

[late parrot]

Let's say currently there is a test build of a project which calls OpenAI. It then calls a library that indirectly calls OpenAI, which will decide on a series of documents (html, pdfs) to download from the internet and then does some logicking process based on the corpus to derive an analysis (let's assume it is a .doc or .md, doesn't really matter here).

The current process is inefficient because

• it downloads the 'corpora' into memory, so the entire process just gets killed off if there is an error
• the library being used is a bit opaque and there is a need to inject some observability tooling (traces, metrics, logs) to see where it is 'choking'

Need some suggestions on how you would proceed in this scenario, or any advice

[flat osprey]

@late parrot

Python help channel opened

Remember to:

• Ask your Python question, not if you can ask or if there's an expert who can help.
• Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
• Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

Closes after a period of inactivity, or when you send !close.

[bitter sail]

if holding the downloaded data in memory is a problem you can always write it to disk
the tempfile module is perfect for this, you can for example use the tempfile.TemporaryDirectory() to create a temporary directory for each task or batch of files that you are working with
if you need to work with the data as if you had it in memory there is also the mmap module to help you with that
both of which are available from the python standard library that every standard python installation should come with

oh, and for paths and general file system stuff i like using the Path class from the pathlib module which is also part of the python standard library
at the end of the pathlib documentation page is a very good cheat sheet of pathlib equivalents (compared to the os module) for many common operations

[flat osprey]

tempfile

Source code: Lib/tempfile.py

This module creates temporary files and directories. It works on all supported platforms. TemporaryFile, NamedTemporaryFile, TemporaryDirectory, and SpooledTemporaryFile are high-level interfaces which provide automatic cleanup and can be used as context managers. mkstemp() and mkdtemp() are lower-level functions which require manual cleanup.

Similar names: anyio.tempfile

mmap

Availability: not WASI.

This module does not work or is not available on WebAssembly. See WebAssembly platforms for more information.

Memory-mapped file objects behave like both bytearray and like file objects. You can use mmap objects in most places where bytearray are expected; for example, you can use the re module to search through a memory-mapped file. You can also change a single byte by doing obj[index] = 97, or change a subsequence by assigning to a slice: obj[i1:i2] = b'...'. You can also read and write data starting at the current file position, and seek() through the file to different positions.

pathlib

Added in version 3.4.

Source code: Lib/pathlib/

This module offers classes representing filesystem paths with semantics appropriate for different operating systems. Path classes are divided between pure paths, which provide purely computational operations without I/O, and concrete paths, which inherit from pure paths but also provide I/O operations.

...

[flat osprey]

Python help channel closed for inactivity

This help channel has been closed. Feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.