Long time to add each item to collection | Kometa | Page 1

coarse night Jul 4, 2023, 9:13 PM

#

Running latest PMM nightly, latest Plex for Windows (x64). I reinstalled Plex today, and only added my TV Shows. It is taking 14 seconds to add each item to a collection, and I do not know what else to do to improve this performance.

I am going collection by collection, commenting out the ones that have been done before, and each time I stop PMM, optimize the Plex DB, restart Plex, and then run PMM again. I have increased the Plex DB Cache size to 2000 MB.

Has anyone else experienced something like this? Any ideas on what else I could do? The Plex machine is a Core i9 with 32 GB RAM, and the Plex DB is on a 1TB SSD. The machine is sitting at around 15% CPU utilization and 25% RAM utilization.

📎 meta.log

supple bisonBOT Jul 4, 2023, 9:13 PM

#

Welcome!

Someone from <@&938443185347244033> will assist when they're available.

Including the meta.log from the beginning is a huge help, type !logs for more information.

#

You can press the "Close Post" button above or type !close at any time to close this post.

supple bisonBOT Jul 4, 2023, 9:13 PM

#

supple bison You can press the "Close Post" button above or type `!close` at any time to clos...

turbid zephyr Jul 5, 2023, 3:37 AM

#

I can't reproduce this behavior on my setup:

| Loading All Movies from Library: Movies - 4K                                                       |
| Loaded 5430 Movies                                                                                 |
|======================================= Movies - 4K Summary ========================================|
|        Title         |   +   |   =   |   -   | Run Time | Status                                   |
|======================|=======|=======|=======|==========|==========================================|
| Netflix Top 50       |    23 |     0 |     0 |  0:00:32 | Created and Updated Metadata, Image      |
| Disney+ Top 50       |    34 |     0 |     0 |  0:01:21 | Created and Updated Metadata, Image      |
| Max Top 50           |    45 |     0 |     0 |  0:07:58 | Created and Updated Metadata, Image      |
| Hulu Top 50          |     7 |     0 |     0 |  0:00:05 | Created and Updated Metadata, Image      |
| Paramount+ Top 50    |    39 |     0 |     0 |  0:03:04 | Created and Updated Metadata, Image      |
| Prime Video Top 50   |    38 |     0 |     0 |  0:00:46 | Created and Updated Metadata, Image      |
| Apple TV+ Top 50     |    17 |     0 |     0 |  0:01:48 | Created and Updated Metadata, Image      |
|                                                                                                    |
| Loading All Shows from Library: TV Shows - 4K                                                      |
| Loaded 1593 Shows                                                                                  |
|====================================== TV Shows - 4K Summary =======================================|
|        Title         |   +   |   =   |   -   | Run Time | Status                                   |
|======================|=======|=======|=======|==========|==========================================|
| Netflix Top 50       |    15 |     0 |     0 |  0:00:40 | Created and Updated Metadata, Image      |
| Disney+ Top 50       |     9 |     0 |     0 |  0:00:37 | Created and Updated Metadata, Image      |
| Max Top 50           |    18 |     0 |     0 |  0:00:33 | Created and Updated Metadata, Image      |
| Hulu Top 50          |     2 |     0 |     0 |  0:00:04 | Created and Updated Metadata, Image      |
| Paramount+ Top 50    |    19 |     0 |     0 |  0:00:28 | Created and Updated Metadata, Image      |
| Prime Video Top 50   |    19 |     0 |     0 |  0:00:29 | Created and Updated Metadata, Image      |
| Apple TV+ Top 50     |    49 |     0 |     0 |  0:00:41 | Created and Updated Metadata, Image      |

One of those takes about 10 seconds each on average, but they seem mostly much faster than that. I deleted all collections at the start of the run, so these were all created from scratch.

        `:+ssssssssssssssssss+:`           ---------
      -+ssssssssssssssssssyyssss+-         OS: Ubuntu 20.04.6 LTS x86_64
    .ossssssssssssssssssdMMMNysssso.       Kernel: 6.1.12-x64v1-xanmod1
   /ssssssssssshdmmNNmmyNMMMMhssssss/      Uptime: 7 days, 12 hours, 5 mins
  +ssssssssshmydMMMMMMMNddddyssssssss+     Packages: 973 (dpkg), 6 (snap)
 /sssssssshNMMMyhhyyyyhmNMMMNhssssssss/    Shell: zsh 5.8
.ssssssssdMMMNhsssssssssshNMMMdssssssss.   Resolution: 1024x768
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   Terminal: /dev/pts/5
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   CPU: Intel i9-10900K (20) @ 5.300GHz
ossyNMMMNyMMhsssssssssssssshmmmhssssssso   GPU: Intel Device 9bc5
+sssshhhyNMMNyssssssssssssyNMMMysssssss+   Memory: 5903MiB / 31755MiB

Plex appdata is on NVME. These are two libraries from a much larger Plex server.

coarse night Jul 5, 2023, 10:39 AM

#

Thanks. Any suggestions ?

quiet notch Jul 5, 2023, 12:45 PM

#

Windows performance can get tricky at times. Bump up Plex db cache to 10Gb. Ensure that antivirus is not scanning the PMM and Plex folders in any way, shape, or form. No file/folder encryption or compression on the two folders. When PMM and Plex are running, review task manager and see what are the biggest users of cpu and memory.

dusky mossBOT Jul 5, 2023, 12:45 PM

#

turbid zephyr Jul 5, 2023, 12:50 PM

#

That SQlite cache setting is now available in the Plex UI.

quiet notch Jul 5, 2023, 12:53 PM

#

We should update that cakeybot message to reflect that

turbid zephyr Jul 5, 2023, 1:10 PM

#

Is PMM running on the same machine as Plex?

dusky mossBOT Jul 5, 2023, 1:12 PM

#

PMM Performance Tips

Use PMM caching where possible: https://metamanager.wiki/en/latest/search.html?q=cache&check_keywords=yes&area=default

Run PMM after PLEX scheduled tasks so 5am is a good time for that since PLEX runs scheduled tasks between 3-5am

Consider Database Cache Size (MB) settings in PLEX under Settings, Library, Database Cache Size (MB). 2GB = 2000MB and 10GB = 10000MB

coarse night Jul 5, 2023, 4:51 PM

#

turbid zephyr Is PMM running on the same machine as Plex?

Yes, they are both on the same machine

coarse night Jul 5, 2023, 4:53 PM

#

quiet notch Windows performance can get tricky at times. Bump up Plex db cache to 10Gb. Ensu...

Thanks. I will change the setting, and add the PMM and Plex folders as exclusions for the built-in antivirus. Python and Plex are the biggest users of cpu and memory when PMM is running.

turbid zephyr Jul 5, 2023, 4:53 PM

#

OK, so not network latency.

coarse night Jul 5, 2023, 7:48 PM

#

@quiet notch , @turbid zephyr I have made the suggested changes, but the performance is exactly the same. It takes 14 seconds to apply an overlay for each episode. At that rate, it will take over 5 days for my 27k episodes to have overlays. Any other ideas?

turbid zephyr Jul 5, 2023, 7:55 PM

#

Not really. Could be some Windows-specific nonsense, but for the most part these things are "hey plex, do this" and then waiting for Plex to return.

#

I'll put some overlays on a large library to see what sort of performance I see on this all-linux setup.

coarse night Jul 5, 2023, 7:57 PM

#

Thanks, it will be good to have a reference

turbid zephyr Jul 5, 2023, 8:03 PM

#

Running your config on:

quiet notch Jul 5, 2023, 8:06 PM

#

I wonder if -lr option could give us more insight to the reason it is taking 14 seconds.

#

-lr is —log-requests

#

https://metamanager.wiki/en/latest/home/environmental.html#log-requests

turbid zephyr Jul 5, 2023, 8:15 PM

#

Perhaps. Each one of those is probably at most two requests, so I suppose that would show if it was the art or the label.

coarse night Jul 5, 2023, 8:16 PM

#

I have been looking at the Plex log, and I can see 13 seconds of these type of messages between each overlay update:

#

Jul 05, 2023 15:50:49.071 [9576] DEBUG - [Req#6da8f] PlayQueue: Converted 'server://88560b6fe82249ebe91e3e91ae012bcb52e36dbf/com.plexapp.plugins.library/library/sections/1/all?type=2&sort=originallyAvailableAt%3Adesc&push=1&show.actor=2604&pop=1' to 'library://x/directory/%2Flibrary%2Fsections%2F1%2Fall%3Ftype%3D2%26sort%3DoriginallyAvailableAt%253Adesc%26push%3D1%26show%2Eactor%3D2604%26pop%3D1'
Jul 05, 2023 15:50:49.072 [9576] DEBUG - [Req#6da8f] Setting container serialization range to [0, 0] (total=-1)

#

I have no idea about what that is. Do you?

coarse night Jul 5, 2023, 8:22 PM

#

turbid zephyr Perhaps. Each one of those is probably at most two requests, so I suppose that ...

I will run the overlays again (I interrupted it) with the lr option

turbid zephyr Jul 5, 2023, 8:22 PM

#

coarse night I have no idea about what that is. Do you?

Not offhand, but I don't know why setting art or a label would be manipulating a "PlayQueue".

coarse night Jul 5, 2023, 8:24 PM

#

I do not know either. I am going to post about this in the Plex forums

turbid zephyr Jul 5, 2023, 8:25 PM

#

I will note that, while watching this run, cataloging all the "missing" things is taking a great deal of time [I typically have that turned off].

#

For example, it took 20 seconds to add 70 shows to the Disney+ collection, then about 2 minutes to log the 294 missing ones.

coarse night Jul 5, 2023, 8:30 PM

#

I turned off displaying that, but it did not make much of a difference

turbid zephyr Jul 5, 2023, 8:31 PM

#

Well, not on the individual add-to-collection time, but it's extending the total run time.

coarse night Jul 5, 2023, 9:14 PM

#

[2023-07-05 17:10:39,594] [connectionpool.py:245] [DEBUG] | Starting new HTTP connection (1): localhost:32400 |
[2023-07-05 17:10:39,609] [connectionpool.py:473] [DEBUG] | http://localhost:32400 "GET /library/metadata/44423/thumb/1688561343?X-Plex-Token=xxxxxxxxxxxxxxx HTTP/1.1" 200 176313 |
[2023-07-05 17:10:39,705] [TiffImagePlugin.py:881] [DEBUG] | Tag 1212, Type: 2, Value: 'overlay' |
[2023-07-05 17:10:39,706] [TiffImagePlugin.py:903] [DEBUG] | save: unknown (1212) - type: string (2) - value: ('overlay',) |
[2023-07-05 17:10:39,706] [TiffImagePlugin.py:930] [DEBUG] | 1212 2 8 b'\x00\x00\x00\x1a' b'overlay\x00' |
[2023-07-05 17:10:39,717] [connectionpool.py:473] [DEBUG] | http://localhost:32400 "POST /library/metadata/44423/posters HTTP/1.1" 200 0 |
[2023-07-05 17:10:53,305] [connectionpool.py:473] [DEBUG] | http://localhost:32400 "PUT /library/sections/1/all?id=44423&label.locked=1&label%5B0%5D.tag.tag=Overlay&type=2 HTTP/1.1" 200 0 |
[2023-07-05 17:10:53,305] [overlays.py:351] [INFO] | Overlays Applied: Overlay File (0) Rating1Rotten, Overlay File (0) Rating2Rotten, Overlay File (0) Rating3Fresh, Overlay File (2) returning_shows |
[2023-07-05 17:10:53,318] [connectionpool.py:473] [DEBUG] | http://localhost:32400 "GET /library/metadata/49609?includePreferences=1 HTTP/1.1" 200 3082

#

This is what I am seeing in the log. I replaced the token with xxxx

turbid zephyr Jul 5, 2023, 9:34 PM

#

Given that these log lines contain return codes, it seems like the log lines follow the call.

Gets the current art, which comes back in 15 milliseconds:

[2023-07-05 17:10:39,594] [connectionpool.py:245]     [DEBUG]    | Starting new HTTP connection (1): localhost:32400                                                  |
[2023-07-05 17:10:39,609] [connectionpool.py:473]     [DEBUG]    | http://localhost:32400/ "GET /library/metadata/44423/thumb/1688561343?X-Plex-Token=xxxxxxxxxxxxxxx HTTP/1.1" 200 176313 |

and checks if it came from PMM which takes no time

[2023-07-05 17:10:39,705] [TiffImagePlugin.py:881]    [DEBUG]    | Tag 1212, Type: 2, Value: 'overlay'                                                                |
[2023-07-05 17:10:39,706] [TiffImagePlugin.py:903]    [DEBUG]    | save: unknown (1212) - type: string (2) - value: ('overlay',)                                      |
[2023-07-05 17:10:39,706] [TiffImagePlugin.py:930]    [DEBUG]    | 1212 2 8 b'\x00\x00\x00\x1a' b'overlay\x00'                                                        |

Then sets the new art, which is back in 10 milliseconds

[2023-07-05 17:10:39,717] [connectionpool.py:473]     [DEBUG]    | http://localhost:32400/ "POST /library/metadata/44423/posters HTTP/1.1" 200 0                       |

Then sets the label, which is back in about 14 seconds:

[2023-07-05 17:10:53,305] [connectionpool.py:473]     [DEBUG]    | http://localhost:32400/ "PUT /library/sections/1/all?id=44423&label.locked=1&label%5B0%5D.tag.tag=Overlay&type=2 HTTP/1.1" 200 0 |

coarse night Jul 5, 2023, 9:37 PM

#

So setting the label is where Plex is taking a long time to respond. This might be an issue with the version of Plex I am using?

#

For testing purposes, how would I go about removing the label setting part of the overlay update?

turbid zephyr Jul 5, 2023, 9:39 PM

#

coarse night For testing purposes, how would I go about removing the label setting part of th...

You can't, at least not without editing the source.

coarse night Jul 5, 2023, 9:41 PM

#

I thought that would be the case. Thanks

turbid zephyr Jul 5, 2023, 9:43 PM

#

PMM uses that to answer "Does this thing have an overlaid image applied?"

coarse night Jul 5, 2023, 9:44 PM

#

Thanks, that makes sense.

turbid zephyr Jul 5, 2023, 9:45 PM

#

Although judging by those timings it may make sense to just download the image to check.

coarse night Jul 5, 2023, 9:46 PM

#

I am not following, what should I do?

turbid zephyr Jul 5, 2023, 9:47 PM

#

Oh, nothing for you to do, just idle chatter about the way the code works. If it takes an age to set the label and a few milliseconds to download and check the image, the label might be the Wrong Answer.

coarse night Jul 5, 2023, 9:48 PM

#

Ah, I get it now.

#

If you had not had this issue before with any of your Windows users, do you think it makes sense to try to uninstall Plex and reinstall an older version to see if that makes a difference?

turbid zephyr Jul 5, 2023, 9:54 PM

#

I personally don't know that it is or isn't a common Windows thing. You could try an older version, but I don't have a specific one to suggest. My linux machine is an older version, as there is an upgrade just above this one that is a one-way valve WRT metadata changes [can't back down after the metadata gets touched with the newer version] and that one introduced other problems so I've stayed where I am [Version 1.29.2.6364].

#

my run is in the Ls settings ratings

coarse night Jul 5, 2023, 9:57 PM

#

When setting ratings for shows and episodes, the run took 1 hour something. I am not setting IMDb labels

#

I thought that was pretty good for 27k episodes

coarse night Jul 5, 2023, 9:58 PM

#

turbid zephyr I personally don't know that it is or isn't a common Windows thing. You could t...

That version works well for all the PMM functionality, correct? I might just try that

turbid zephyr Jul 5, 2023, 10:45 PM

#

Works fine.

Into overlay filtering.

coarse night Jul 5, 2023, 11:38 PM

#

Thanks. Reinstalled Plex, the same version you are using. I will only process overlays for tv shows (not seasons or episodes) to see how it goes. I am also at overlay filtering

#

So far, so good! It seems to be the Plex version. The overlays for tv shows are being updated at a rate of 5 per second now 🙂

#

Finished. Total time for overlays (including filtering, etc.) was 13 minutes. Now I will do ratings updates and overlays for episodes.

quiet notch Jul 5, 2023, 11:51 PM

#

Wow… I am perplexed at how the version has affected the performance like that.

coarse night Jul 5, 2023, 11:53 PM

#

Let's see what happens with episodes (27k)

quiet notch Jul 5, 2023, 11:54 PM

#

5/sec sounds better than 1/15sec

coarse night Jul 5, 2023, 11:55 PM

#

Very true, and that includes setting the label "Overlay"

turbid zephyr Jul 6, 2023, 12:03 AM

#

Still filtering for overlays here.

quiet notch Jul 6, 2023, 12:20 AM

#

Hmm… if Plex version changes performance of PMM, might be good to grab the Plex version (through the PlexAPI I presume) and report that at the top with the other PMM information in the meta.log.

coarse night Jul 6, 2023, 11:10 AM

#

Hello. My run finished in 4 hours and 50 minutes. So yes, the issue appears to be the Plex version

#

I will now finally do a full run of my TV library

quiet notch Jul 6, 2023, 11:16 AM

#

As a student Mary of Plex version situation, what version were you on and which did you end up on?

coarse night Jul 6, 2023, 11:43 AM

#

I was on the latest, 1.32.5.7210. I ended up using the same one @turbid zephyr is on, 1.29.2.6364

#

I suppose there are newer versions between those two that work, but it would be cumbersome for me to try and test to see where this issue first pops up.

#

My post in the Plex support forums has not received a reply yet

coarse night Jul 6, 2023, 6:09 PM

#

Wow! My first full run of TV Shows 🙂 6 hous, 53 minutes. Thank you @turbid zephyr and @quiet notch

quiet notch Jul 7, 2023, 12:45 AM

#

coarse night My post in the Plex support forums has not received a reply yet

Typical

coarse night Jul 7, 2023, 2:15 PM

#

Unfortunately, the Plex database got corrupted after the movies library PMM run finished. I could not repair it, so I will have to start from scratch again

quiet notch Jul 8, 2023, 1:46 AM

#

Any updates?

coarse night Jul 8, 2023, 5:08 PM

#

Sorry, yes, I added all my movies and was able to do a complete PMM run of this library. It took 2 hours, 42 minutes, so performance is fine for my 9331 movie collection. A little later today I will add the tv shows again and do a PMM run. Tomorrow I will have a further update.

forest acorn Jul 10, 2023, 2:42 PM

#

can this be closed?

#Long time to add each item to collection