#Memory usage is quite high

1 messages Β· Page 1 of 1 (latest)

brave ice
#

PMM is using 2.4GB of memory, which is way too much for a raspberry pi.
Why does it need that much memory?

I am using Docker in linux and working on 4 libraries.

image.png
whole lagoonBOT
#

Please complete #938455615741775902 and someone from the community or one of our <@&938443185347244033> members will respond when they're available.

Including the meta.log from the beginning is a huge help use !logs for more information.

brave ice
ripe steppe
#

overlays are quite resource intensive in particular

brave ice
#

i think i can handle it by running during the night and using swap, but i wonder if the program could be optimized to use less memory

glacial mauve
#

It could be; there's been little memory-specific tuning.

#

It's a matter of someone knowledgeable in the area devoting time to look at it.

river spruce
#

Im not very great with memory optimization practices

brave ice
#

i can try having a look at it

brave ice
#

it seems that most of the memory is used on saving the overlays in memory

#

since the method of operation if i understood correctly is to:

  • produce all overlays for all collections
  • apply all overlays to all collections
#

this means that all images are stored in memory until they are applied, which is not very memory efficient

#

it gets worse as the plex server is bigger

river spruce
#

the overlay itself is all that should be stored

#

not the underlying poster

#

and then it should all get combined when applying

#

so it should be number of overlays is whats stored not number of items in your library]\

#

at least thats the intention

#

very possible i fucked it up somewhere

brave ice
#

i dont know, if you check the html i posted (flamegraph of memory allocations) there are 384 + 380 + 380 allocations of images

#

that is a lot

#

πŸ˜„

#

maybe something that can be optimized

river spruce
#

ok 2 things

#

#1 what program did you use to create the above html file

#

#2 can you try again with the latest nightly build i just pushed nightly140

#

@brave ice

brave ice
#

i also had to do a slight tweak since pmm spawns a new python process by default which is a bit weird and these profiling tools do not work well with multiple processes

#

i can try with nightly yes

river spruce
brave ice
#

i see

#

anyways it is a simple patch just to make the profiling work properly

#

to generate the flamegraph:
memray flamegraph memray-plex_meta_manager.py.208534.bin
(filename will change depending on the process id)

river spruce
#

so yea all that memory waste was due to backdrops being stored in memory which ive removed and they should now be generated on the fly

brave ice
#

nice! i will test

#

and get back to you soon

#

ok seems to be fixed, in terms of memory usage

#

much better!

river spruce
#

nice

brave ice
#

i am getting some errors though. not sure if i was getting them before:

| Perfect Blue                                 |
[2023-03-16 18:56:31,850] [overlays.py:362]           [INFO]     | cannot determine region size; use 4-item box                                                       |
[2023-03-16 18:56:31,851] [overlays.py:363]           [INFO]     | <class 'ValueError'>                                                                               |
#

seems to be new, since my old logs dont have it

river spruce
#

can you send me the full log

jaunty boughBOT
#
Providing Log Files

If you are not already in one, open a thread in #pmm-help

Find meta.log in config/logs/

Drag it in the Discord thread - the whole thing, DO NOT EDIT IT

Log file too big? USE PASTEBIN or ZIP THE FILE

Still confused? Β―\(ツ)/Β―

Thanks for using Plex Meta Manager!
brave ice
#

yes

#

it exited early because i interrupted it since i was seeing errors

#

the error has little information but seems to be related to image code

#

btw, it seems that memray supports tracking memory with multiple processes but it needs to be explicitly defined:
memray run --follow-fork example.py (i did not test, was just curious and looked it up)

river spruce
#

line 364 in overlays.py can you change logger.stacktrace(e) to logger.stacktrace() (remove the e) and then run again so i have the stacktrace?

brave ice
#

ye

river spruce
#

ty

brave ice
#

np

river spruce
#

just before line 313 in the same file can you make a new line and add logger.info(overlay_image)

ripe steppe
river spruce
#

change 312 to if overlay_image is not None

#

and then run again

ripe steppe
#

on it

river spruce
#

looks like that worked

ripe steppe
#

I'll re-enable cache and do it a couple times to make sure it doesn't reapply

#

all good πŸ™‚

river spruce
#

word

ripe steppe
#

We should ask that guy to re-test this

#

If I can hunt him out

glacial mauve
river spruce
#

which guy

ripe steppe
#

The guy who suggested closing the images in 2 places

glacial mauve
#

Oh I think he abandoned PMM and wrote his own overlay script that takes negative time and memory

ripe steppe
#

omg

#

hahahahaha

#

I thought that was someone else lmao

glacial mauve
#

Can't recall, just bringing the COMEDY

ripe steppe
#

Do you use overlays carrot?

#

you'd be a good guinea pig given the size of your library πŸ˜„

glacial mauve
#

I can give it a try.

#

I usually run in docker though.

ripe steppe
#

ah right

glacial mauve
#

I can try the native install on this machine.

ripe steppe
#

so you can't do that memthing

brave ice
#

i can try it again soon, nightly branch?

#

my library is not very big though

ripe steppe
#

yes nightly πŸ™‚

brave ice
#

seems to be working

#

atleast the error is not showing anymore, it is now saying "overlay applied (...)" which seems good

#

in terms of memory it also seems good

#

going to let the script finish executing, as it is still running

brave ice
#

ok it has finished with no problems