I have a Discord bot that does image generation using the PIL library.
To save image opening time, I store the opened image file in memory with the help of cachetools.TTLCache.
But after doing so, I've been seeing some errors, such as file pointer issues (like reading the image while caching at the same time? not exactly sure what happened). So I added a threading.Lock() to the lock parameter of TTLCache.
And now, I see a new error that sometimes occur when opening images, where the file pointer will become NoneType, I asked gpt and it says it might be my caching system that's causing the issue.
So I'm thinking that, maybe I am on the wrong track? Am I not supposed to cache opend image objects?
#🔒 Is caching opened image file objects a bad practice?
upbeat hamlet
sleek marlinBOT
I have a Discord bot that does image generation using the PIL library.
To save i...
@upbeat hamlet
Python help channel opened
upbeat hamlet
code of caching implementation
deco = cachetools.cached(cachetools.TTLCache(maxsize=128, ttl=300), lock=threading.Lock())
image_read = deco(Image.open)
def open_image(file_path: pathlib.Path | str, size: tuple[int, int] | None = None) -> Image.Image:
image = image_read(file_path)
image = image.convert("RGBA")
if size is not None:
image = image.resize(size, Image.Resampling.LANCZOS)
return image.copy()
fallen plinth
code of caching implementation
```py
deco = cachetools.cached(cachetools.TTLCach...
Pillow does not actually loads into memory when you call Image.open,
from https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.open
This is a lazy operation; this function identifies the file, but the file remains open and the actual image data is not read from the file until you try to process the data (or call the load() method)
upbeat hamlet
seems to be true...
from PIL import Image
import time
for _ in range(5):
start = time.perf_counter()
Image.open("hoyo_buddy/web_app/assets/images/dev_tools_tutorial.gif")
print(time.perf_counter() - start)
0.02123070000379812
0.0002891999974963255
0.0002630000017234124
0.00019439999596215785
0.00024839999969117343
acoustic flint
this documentation seems relevant as well:
https://pillow.readthedocs.io/en/stable/reference/open_files.html#image-lifecycle
When the pixel data from the image is required,
Image.load()is called. The current frame is read into memory. The image can now be used independently of the underlying image file.The lifecycle of a single-frame image is relatively simple. The file must remain open until the
load()orclose()function is called or the context manager exits.
upbeat hamlet
i've never used the load() or close() methods tho, i only use Image.open
acoustic flint
i'd assume once you've called load(), you've saved the image in memory and can make as many copies of it as you want
upbeat hamlet
so it only closes the image when close() is called? (or when it is gced)
by closes the image i mean offloading it from mem
acoustic flint
close() is different from the context manager in that not only will it close the file, but also clean up the image data from memory
upbeat hamlet
hmm, i see
acoustic flint
as per that same image lifecycle doc:
Image.Image.close()Closes the file and destroys the core image object.The Pillow context manager will also close the file, but will not destroy the core image object. e.g.:
with Image.open("test.jpg") as img: img.load() assert img.fp is None img.save("test.png")
upbeat hamlet
but what i want to do here is let the image file retain longer in the memory (around 5 minutes) so that future image opening of the same file can take the opened image file directly from mem (faster)
fallen plinth
I would try something like ```py
@cachetools.cached(cachetools.TTLCache(maxsize=128, ttl=300), lock=threading.Lock())
def image_read(path):
with Image.open("test.jpg") as img:
img.load()
return img.copy()
not sure if that'll work though, specially if you are dealing with gifs
pulsar lichen
I would expect that all you need is to add a .load() call into your open_image(). And maybe don't do copy()?
in theory after a load it should be in memory.
upbeat hamlet
I would try something like ```py
@cachetools.cached(cachetools.TTLCache(maxsize=...
btw, im running this inside async context (the file opening is sync, but i run it inside an executor), does this matter or nah
pulsar lichen
interestingly:
The lock context manager is used only to guard access to the cache object. The underlying wrapped function will be called outside the with statement, and must be thread-safe by itself.
so currently, you can potentially get two concurrent image reads. this is probably only a problem if they're to the same file, though, and maybe not even then
upbeat hamlet
yeah, i see why (the error happend) now
i'm gonna remove the caching implementation for now, the performance benefit it brings doesn't match up with the trouble it can potentially have
i prefer reliability
thanks for the helps
sleek marlinBOT
Python help channel closed