Kometa taking 20+ hours to complete | Kometa | Page 1

tribal grotto Nov 1, 2024, 2:35 PM

#

Hello, just looking for a little guidance, over the last 30-45 days or so Kometa is taking upwards for 20+ hours to complete.

lament impBOT Nov 1, 2024, 2:35 PM

#

Welcome @tribal grotto!

Someone from <@&938443185347244033> will assist when they're available.

Including meta.log from the beginning is a huge help. Type !logs for more information.

After attaching your log, do not forget to hit the green check boxes when prompted by our bot.

#

You can press the "Close Post" button above or type /close at any time to close this post.

lament impBOT Nov 1, 2024, 2:35 PM

#

lament imp You can press the "Close Post" button above or type `/close` at any time to clos...

tribal grotto Nov 1, 2024, 2:35 PM

#

📎 meta.log

lament impBOT Nov 1, 2024, 2:35 PM

#

📝 If you want to review this again, topochocolate:
:one: Right-click (or long press with phone) on the message that contains the log
:two: Select: Copy Message Link
:three: Use the command: /logscan <message_link> or !logscan <message_link> and paste the value copied from the previous step where you see <message_link> 📝

#

📝 If you want to review this again, topochocolate:
:one: Right-click (or long press with phone) on the message that contains the log
:two: Select: Copy Message Link
:three: Use the command: /logscan <message_link> or !logscan <message_link> and paste the value copied from the previous step where you see <message_link> 📝

#

📝 If you want to review this again, topochocolate:
:one: Right-click (or long press with phone) on the message that contains the log
:two: Select: Copy Message Link
:three: Use the command: /logscan <message_link> or !logscan <message_link> and paste the value copied from the previous step where you see <message_link> 📝

tribal grotto Nov 1, 2024, 2:38 PM

#

!logscan #1301917446411915345 message

lament impBOT Nov 1, 2024, 2:38 PM

#

📝 If you want to review this again, topochocolate:
:one: Right-click (or long press with phone) on the message that contains the log
:two: Select: Copy Message Link
:three: Use the command: /logscan <message_link> or !logscan <message_link> and paste the value copied from the previous step where you see <message_link> 📝

#

💥An error occurred while processing the attachment.💥

steady prism Nov 1, 2024, 2:50 PM

#

Well, the high-level view:

|========================================== Movies Summary ==========================================|
|
| Title                       | Run Time |
| =========================== | ======== |
| Library Loading and Mapping |  0:01:00 |
| Library Operations          |  0:00:01 |
| Library Images Files        |  0:00:00 |
| Library Metadata Files      |  0:00:00 |
| Library Collection Files    |  4:48:22 |
| Library Overlay Files       | 1 day, 2:57:58 |
|
|========================================= TV Shows Summary =========================================|
|
| Title                       | Run Time |
| =========================== | ======== |
| Library Loading and Mapping |  0:00:15 |
| Library Operations          |  0:00:03 |
| Library Images Files        |  0:00:00 |
| Library Metadata Files      |  0:00:00 |
| Library Collection Files    |  0:06:29 |
| Library Overlay Files       |  0:19:56 |

It's almost entirely overlays in the movie library.

tribal grotto Nov 1, 2024, 2:51 PM

#

i mean, i get that there's 6300 movies

#

however i swear at 6200 movies this was only taking 2 hrs

steady prism Nov 1, 2024, 2:52 PM

#

There don't seem to be all that many:

| Reading in Overlay Files                                                                           |
|                                                                                                    |
| Reading url: https://raw.githubusercontent.com/TheChrisK/PMM/main/overlays/Background.yml          |
|                                                                                                    |
| Reading default: resolution                                                                        |
| Template Variables: {'url': 'https://raw.githubusercontent.com/TheChrisK/PMM/main/overlays/resolution-top-left-45deg/<<overlay_name>>.png', 'horizontal_align': 'left', 'horizontal_offset': 0, 'vertical_offset': 0, 'vertical_align': 'top', 'final_horizontal_offset': 0, 'final_vertical_offset': 0, 'back_width': 1000, 'back_height': 1500, 'back_color': 0} |
|                                                                                                    |
| Reading default: audio_codec                                                                       |
| Template Variables: {'url': 'https://raw.githubusercontent.com/TheChrisK/PMM/main/overlays/audio-top-left-45deg/<<key>>.png', 'horizontal_align': 'left', 'horizontal_offset': 0, 'vertical_offset': 0, 'vertical_align': 'top', 'back_width': 1000, 'back_height': 1500, 'back_color': 0} |
|                                                                                                    |
| Reading default: ribbon                                                                            |

tribal grotto Nov 1, 2024, 2:54 PM

#

this was 10/15

📎 meta9.log

lament impBOT Nov 1, 2024, 2:54 PM

#

📝 If you want to review this again, topochocolate:
:one: Right-click (or long press with phone) on the message that contains the log
:two: Select: Copy Message Link
:three: Use the command: /logscan <message_link> or !logscan <message_link> and paste the value copied from the previous step where you see <message_link> 📝

#

**User Info**

Author of Linked Message: topochocolate
Person who Invoked the Command: topochocolate
File Name: meta9.log

Table of Contents:
Page 01: User Info
Page 02: Kometa Info
Page 03: Kometa Summary Info
Page 04: Kometa Config.yml YAML Validation ✅
Page 05: Plex Configuration - Section 1
Page 06: Plex Configuration - Section 2
Page 07: Rec 01 - 🚀 VERSION UPDATE AVAILABLE
Page 08: Rec 02 - ❌ ANIDB AUTH ERRORS
Page 09: Rec 03 - ❌🔒 BLANK API KEY ERROR
Page 10: Rec 04 - ❌ [ERROR]
Page 11: Rec 05 - ❌ MY ANIME LIST CONNECTION ERROR
Page 12: Rec 06 - ❌ TRAKT CONNECTION ERROR
Page 13: Rec 07 - ⚠️ [WARNING]
Page 14: Rec 08 - ⚠️ NO ITEMS FOUND IN PLEX
Page 15: Rec 09 - 💬 CONVERT WARNING

steady prism Nov 1, 2024, 2:55 PM

#

Whatever this overlay does is taking 11 hours by itself:

📎 message.txt

tribal grotto Nov 1, 2024, 2:56 PM

#

hmmmm

#

i think that's literally just a black triangle that the text sits over

#

#

and it's saying that zero of the 6300 movies have it applied already?

steady prism Nov 1, 2024, 2:59 PM

#

Do you mean the (0)? No, that's just an index. Not a statement of how many have it.

tribal grotto Nov 1, 2024, 2:59 PM

#

oh i see

steady prism Nov 1, 2024, 3:00 PM

#

plex_all returns 6332 items:

[2024-10-29 17:07:06,189] [builder.py:2233]           [DEBUG]    | 6332 IDs Found

Then filtering happens on those 6332 items, which takes 11 hours, and all 6332 items qualify for this overlay

[2024-10-30 04:21:20,709] [overlays.py:587]           [INFO]     | 6332 Items found for Overlay File (0) Background

#

Then all the resolution overlays process in 10 minutes:

[2024-10-30 04:21:20,815] [builder.py:203]            [INFO]     |                             4K-DV-HDR-Plus-Dovetail Overlay in Movies                              |
...
[2024-10-30 04:31:41,393] [overlays.py:590]           [WARNING]  | No Items found for Overlay File (1) Ulysses                                                        |

#

Audio in 2 minutes:

[2024-10-30 04:31:41,394] [builder.py:203]            [INFO]     |                                Dolby-TrueHD-Atmos Overlay in Movies
...

[2024-10-30 04:33:31,222] [overlays.py:587]           [INFO]     | 6 Items found for Overlay File (2) Opus

#

ribbons in one minute:

[2024-10-30 04:33:31,224] [builder.py:203]            [INFO]     |                               Oscars Best Picture Overlay in Movies                                |

[2024-10-30 04:34:51,829] [overlays.py:587]           [INFO]     | 25 Items found for Overlay File (3) Razzies Winner                                                 |

#

Then overlay application, which only actually applies a handful of overlays, takes 16 hours:

[2024-10-30 04:35:35,471] [overlays.py:73]            [INFO]     |                              Applying Overlays for the Movies Library                              |
[2024-10-30 20:04:27,098] [overlays.py:524]           [INFO]     |   Overlay Update Not Needed                                                                        |

#

Those are insane numbers

tribal grotto Nov 1, 2024, 3:06 PM

#

i'm so confused, so why were there 11+ hours of extra processing on the 29th vs the 15th with the same script? only 40'ish movies were added between those dates

#

is my kometa cache corrupt?

steady prism Nov 1, 2024, 3:07 PM

#

I can't imagine the cache causing this; it's a tiny local SQLIte DB.

tribal grotto Nov 1, 2024, 3:08 PM

#

so 10/15 it took 42 minutes to run

#

and then 10/16 it took 16 hours

steady prism Nov 1, 2024, 3:10 PM

#

Now that I look at that background overlay file, it's even more of a mystery. That file has no filtering, only a plex_all: true

So there should be nothing going on during this 11 hour gap:

[2024-10-29 17:07:06,189] [builder.py:2233]           [DEBUG]    | 6332 IDs Found                                                                                     |
[2024-10-30 04:21:20,709] [overlays.py:587]           [INFO]     | 6332 Items found for Overlay File (0) Background                                                   |

#

I guess I would start with removing the cache.

#

Have then been any other changes in the environment at all?

tribal grotto Nov 1, 2024, 3:12 PM

#

not really, i mean, i started replacing some movies with copies that had better audio

#

oh and get this

#

last night i setup Kometa on my desktop, and ran it against Plex from there, so it had never existed on that machine before

#

it's been 12 hours and it's still running

#

so can't really be a kometa cache

#

*issue

cold abyss Nov 1, 2024, 3:14 PM

#

"last night i setup Kometa on my desktop, and ran it against Plex from there, so it had never existed on that machine before"

where was it running from before?

tribal grotto Nov 1, 2024, 3:14 PM

#

directly on the server hosting plex

cold abyss Nov 1, 2024, 3:14 PM

#

A machine with more CPU power?

#

and RAM etc

tribal grotto Nov 1, 2024, 3:15 PM

#

plex server has better CPU but less RAM

steady prism Nov 1, 2024, 3:15 PM

#

How many more variables are there?

hidden ember Nov 1, 2024, 3:15 PM

#

tribal grotto it's been 12 hours and it's still running

I delete my cache yesterday and it took 12 hours to run 2300 movies and 430 shows (with ~13500 episodes). You have 3 times the movies so I'd gues it would take 3x as long?

tribal grotto Nov 1, 2024, 3:15 PM

#

i've always historically run Kometa on the Plex server

#

the machine it's currently executing on is an i7-8700 with 64GBs of RAM

cold abyss Nov 1, 2024, 3:16 PM

#

Try that again and see if you're still hit with the long runtimes?

tribal grotto Nov 1, 2024, 3:16 PM

#

Plex server is like an i5-12450H with 16GBs

tribal grotto Nov 1, 2024, 3:16 PM

#

cold abyss Try that again and see if you're still hit with the long runtimes?

after first initial run on i7-8700 machine?

cold abyss Nov 1, 2024, 3:16 PM

#

If it's running on the same machine perhaps the API requests are also being processed quicker

cold abyss Nov 1, 2024, 3:17 PM

#

tribal grotto after first initial run on i7-8700 machine?

Sure, you could copy over your config.cache and config.yml so that it's using the same cache etc

tribal grotto Nov 1, 2024, 3:17 PM

#

i copied config.yml

cold abyss Nov 1, 2024, 3:17 PM

#

42 minutes up to 16 hours does seem like an insane jump, but it's going to be difficult to know what exactly is causing it

#

if there are a lot of environmental changes happening

tribal grotto Nov 1, 2024, 3:18 PM

#

well like i said the only major change between the 42 mins and 16 hours was just the addition of maybe 40 movies

#

unless something was changed on the git repo my script pulls the overlays from

cold abyss Nov 1, 2024, 3:19 PM

#

The default cache expiration is 60 days I believe, so maybe you've hit that 60 day mark?

tribal grotto Nov 1, 2024, 3:19 PM

#

but doesn't that mean that the day after the 16hr run it would be back to 42 mins'ish?

zealous blaze Nov 1, 2024, 3:20 PM

#

I mean if you moved it and there was Zero cache, that will contribute. I deleted my cache last week and the run took 12+ hours with 2500 movies and 20 shows. When normally my runs take an hour

cold abyss Nov 1, 2024, 3:20 PM

#

Yes I would expect an increase in the performance if things are re-cached

#

I'm going to slightly bow out as I'm thinking there may be too many cooks in the kitchen here and I don't wanna step on toes 😄

steady prism Nov 1, 2024, 3:21 PM

#

There are two places where the time is going:

11 hours here:

[2024-10-29 17:07:06,189] [builder.py:2233]           [DEBUG]    | 6332 IDs Found                                                                                     |
[2024-10-30 04:21:20,709] [overlays.py:587]           [INFO]     | 6332 Items found for Overlay File (0) Background                                                   |

and 16 hours here:

[2024-10-30 04:35:35,471] [overlays.py:73]            [INFO]     |                              Applying Overlays for the Movies Library                              |
[2024-10-30 20:04:27,098] [overlays.py:524]           [INFO]     |   Overlay Update Not Needed                                                                        |

#

Neither of those seems like cache-related things; they seem like file access issues.

tribal grotto Nov 1, 2024, 3:21 PM

#

so something with the git repo?

steady prism Nov 1, 2024, 3:22 PM

#

No, the media files on the plex server.

tribal grotto Nov 1, 2024, 3:22 PM

#

oh

#

that's weird

steady prism Nov 1, 2024, 3:22 PM

#

I'm just guessing. There is no github access during either of those periods.

tribal grotto Nov 1, 2024, 3:22 PM

#

i mean nothing changed with the API

#

i see

#

is my Plex DB cache messed up?

steady prism Nov 1, 2024, 3:23 PM

#

There is no way to tell what specifically is going on with the information we have at hand.

#

Anything is a guess.

tribal grotto Nov 1, 2024, 3:23 PM

#

are there any Plex logs that can help?

steady prism Nov 1, 2024, 3:24 PM

#

There are Plex logs, sure. Whether they help or not is unknown. This situation you are describing is not one that has come up before.

tribal grotto Nov 1, 2024, 3:24 PM

#

ha ha

#

great

cold abyss Nov 1, 2024, 3:25 PM

#

Kometa trace logs would probably be of some assistance

#

it would give us an idea of what's happening during those multi-hour gaps in your runs, but you'd need to enable trace logging and do another run for it to capture

tribal grotto Nov 1, 2024, 3:26 PM

#

oh man

#

ok

#

i guess i gotta wait for the current run to end first

#

what are the flags to enable trace?

cold abyss Nov 1, 2024, 3:26 PM

#

You could try running DBRepair as well which should help make sure your database is in a good state

tribal grotto Nov 1, 2024, 3:26 PM

#

Plex DBrepair?

cold abyss Nov 1, 2024, 3:26 PM

#

you can type !dbrepair to learn more on that tool, TAKE BACKUPS FIRST if you do use it

cold abyss Nov 1, 2024, 3:27 PM

#

tribal grotto what are the flags to enable trace?

python kometa.py -tr -r

waxen barnBOT Nov 1, 2024, 3:27 PM

#

Plex DB Repair Tool

DBRepair provides database repair and maintenance for the most common Plex Media Server database problems. It is a simple menu-driven utility with a command line backend.

DBRepair is run from a command line (terminal or ssh/putty session) which has sufficient privilege to read/write the databases (minimum). If sufficient privleges exist (root), and supported by the environment, the options to start and stop PMS are presented as well. (Some envionments require DBRepair to run as the 'root' user.)

You can learn more about this tool and download/use it on ChuckPA's GitHub Repository https://github.com/ChuckPa/PlexDBRepair

cold abyss Nov 1, 2024, 3:27 PM

#

Okay bowing out again 😄

tribal grotto Nov 1, 2024, 3:30 PM

#

alright so once the curren run of Kometa completes, i'll run the Plex dbrepair and then run kometa again with tracing enable

#

*enabled

#

we're at 1852/6333 movies so it should end in about 5500 minutes 😭

steady prism Nov 1, 2024, 3:34 PM

#

You could just cancel this run.

tribal grotto Nov 1, 2024, 3:34 PM

#

don't think that will mess anything up further?

steady prism Nov 1, 2024, 3:35 PM

#

It typically won't, no.

tribal grotto Nov 1, 2024, 3:39 PM

#

any idea how long the plex dbrepair will take?

steady prism Nov 1, 2024, 3:39 PM

#

It's typically very fast.

tribal grotto Nov 1, 2024, 3:39 PM

#

ok that's good

#

i guess i'll start backing up Plex now

steady prism Nov 1, 2024, 3:41 PM

#

FWIW, I just ran that problematic background overlay on my test library and it flew through, so there isn't something specific to that overlay.

tribal grotto Nov 1, 2024, 3:42 PM

#

got it

#

appreciate that

#

so all signs seem to point to something plex related

#

on my system

#

Plex data ~ 70GBs

#

i feel like...... that's big even for the size of my library

#

metadata folder is 37 GBs alone

cold abyss Nov 1, 2024, 3:51 PM

#

You may have bloat

waxen barnBOT Nov 1, 2024, 3:51 PM

#

Worried about Image Bloat?

Use ImageMaid to get rid of unwanted bloat!

Features include:
:one: Cleaning up of custom posters and title cards that were uploaded to Plex and are not in use anymore
:two: Deleting the Phototranscoder folder to free up even more space
:three: Perform the Empty Trash, Clean Bundles and Optimize DB operations
:four: Scheduling ImageMaid to run regularly

tribal grotto Nov 1, 2024, 3:52 PM

#

seems like a strong possibility

tribal grotto Nov 1, 2024, 4:40 PM

#

ran the DB repair tool

#

i'll try a kometa with trace back on the plex server

hidden ember Nov 1, 2024, 5:17 PM

#

tribal grotto Plex data ~ 70GBs

My plex database is 448MB. My Plex metadata directory is 33.8GB. You have 3 times the movies I do but I think I have 10 times the shows you do.

#

So 70GB seems large

tribal grotto Nov 1, 2024, 5:59 PM

#

kometa still running an hour in

steady prism Nov 1, 2024, 6:01 PM

#

Anything interesting in the log?

tribal grotto Nov 1, 2024, 7:00 PM

#

unsure

#

this is a copy of where it's at currently

#

📎 meta.log

lament impBOT Nov 1, 2024, 7:00 PM

#

📝 If you want to review this again, topochocolate:
:one: Right-click (or long press with phone) on the message that contains the log
:two: Select: Copy Message Link
:three: Use the command: /logscan <message_link> or !logscan <message_link> and paste the value copied from the previous step where you see <message_link> 📝

#

**Kometa Info**

Kometa Header

Version: 2.0.2 (Python 3.11.0) (Git: master)
Newest Version: 2.1.0
Platform: Windows-10-10.0.22631-SP0
Total Memory: 16 GB
Available Memory: 7 GB

Run Command: C:\Users\topo\Kometa\kometa.py -tr -r

Kometa Footer

The footer was empty/not found in meta.log. Go see recommendations pages.

Available Kometa Versions

as of 2024-11-01 20:00:40

Kometa Version (Master branch)

2.1.0

Kometa Version (Develop branch)

2.1.0

Kometa Version (Nightly branch)

2.1.0-build10

#

📎 parsed_meta.log_config_topochocolate.yml

steady prism Nov 1, 2024, 7:01 PM

#

OK, so still working on collections

tribal grotto Nov 1, 2024, 9:28 PM

#

still running

tribal grotto Nov 2, 2024, 2:42 AM

#

still going -_-

tribal grotto Nov 2, 2024, 3:48 AM

#

crawling on overlays again so likely won't finish for 11+ hours

tribal grotto Nov 2, 2024, 1:50 PM

#

alright so it completed, however the log file with trace exceeds the size limit i can upload

#

splitting it into two logs and zipping it to get each under 10 mb

📎 meta1.zip

lament impBOT Nov 2, 2024, 1:59 PM

#

📝 If you want to review this again, topochocolate:
:one: Right-click (or long press with phone) on the message that contains the log
:two: Select: Copy Message Link
:three: Use the command: /logscan <message_link> or !logscan <message_link> and paste the value copied from the previous step where you see <message_link> 📝

tribal grotto Nov 2, 2024, 2:00 PM

#

📎 meta2.zip

lament impBOT Nov 2, 2024, 2:00 PM

#

**User Info**

Author of Linked Message: topochocolate
Person who Invoked the Command: topochocolate
File Name: meta1.zip

Table of Contents:
Page 01: User Info
Page 02: Kometa Info
Page 03: Kometa Summary Info
Page 04: Kometa Config.yml YAML Validation ✅
Page 05: Plex Configuration - Section 1
Page 06: Plex Configuration - Section 2
Page 07: Rec 01 - 🚀 VERSION UPDATE AVAILABLE
Page 08: Rec 02 - ❌ ANIDB AUTH ERRORS
Page 09: Rec 03 - ❌🔒 BLANK API KEY ERROR
Page 10: Rec 04 - ❌ [ERROR]
Page 11: Rec 05 - ❌🛠️ INCOMPLETE LOGS
Page 12: Rec 06 - ❌ MY ANIME LIST CONNECTION ERROR
Page 13: Rec 07 - ❌ TRAKT CONNECTION ERROR
Page 14: Rec 08 - ⚠️ [WARNING]
Page 15: Rec 09 - 💬 CONVERT WARNING

#

📝 If you want to review this again, topochocolate:
:one: Right-click (or long press with phone) on the message that contains the log
:two: Select: Copy Message Link
:three: Use the command: /logscan <message_link> or !logscan <message_link> and paste the value copied from the previous step where you see <message_link> 📝

tribal grotto Nov 2, 2024, 2:01 PM

#

let me know if there's an alternative workaround

lament impBOT Nov 2, 2024, 2:01 PM

#

**User Info**

Author of Linked Message: topochocolate
Person who Invoked the Command: topochocolate
File Name: meta2.zip

Table of Contents:
Page 01: User Info
Page 02: Kometa Info
Page 03: Kometa Summary Info
Page 04: Kometa Config.yml YAML Validation ❌
Page 05: Rec 01 - 🚀 VERSION UPDATE AVAILABLE
Page 06: Rec 02 - ❌ ANIDB AUTH ERRORS
Page 07: Rec 03 - ❌🔒 BLANK API KEY ERROR
Page 08: Rec 04 - ❌ [ERROR]
Page 09: Rec 05 - ❌🛠️ INCOMPLETE LOGS
Page 10: Rec 06 - ⚠️ [WARNING]
Page 11: Rec 07 - ⚠️ NO ITEMS FOUND IN PLEX
Page 12: Rec 08 - 💬 CONVERT WARNING
Page 13: Rec 09 - Error: Plex scheduled time is missing.
Page 14: Rec 10 - Error: Memory value not found in content.

#

📎 parsed_meta1.zip_config_topochocolate.yml

steady prism Nov 2, 2024, 3:06 PM

#

Step one, you're using an older version, so you will want to update:

[INFO]     |     Version: 2.0.2 (Python 3.11.0) (Git: master)                                                   |
[INFO]     |     Newest Version: 2.1.0                                                                          |

#

Still the same things consuming the time:

|========================================== Movies Summary ==========================================|
|                                                                                                    |
| Title                       | Run Time |                                                           |
| =========================== | ======== |                                                           |
| Library Loading and Mapping |  0:00:33 |                                                           |
| Library Operations          |  0:00:01 |                                                           |
| Library Images Files        |  0:00:00 |                                                           |
| Library Metadata Files      |  0:00:00 |                                                           |
| Library Collection Files    |  3:38:24 |                                                           |
| Library Overlay Files       | 15:28:51 |                                                           |
|                                                                                                    |
|========================================= TV Shows Summary =========================================|
|                                                                                                    |
| Title                       | Run Time |                                                           |
| =========================== | ======== |                                                           |
| Library Loading and Mapping |  0:00:08 |                                                           |
| Library Operations          |  0:00:01 |                                                           |
| Library Images Files        |  0:00:00 |                                                           |
| Library Metadata Files      |  0:00:00 |                                                           |
| Library Collection Files    |  0:06:08 |                                                           |
| Library Overlay Files       |  0:13:58 |                                                           |

#

And it's still in the same places:

7 hours to turn 6333 Plex IDs to movies:

[2024-11-01 15:22:27,294] [builder.py:2233]           [DEBUG]    | 6333 IDs Found                                                                                     |
[2024-11-01 15:22:27,296] [builder.py:2234]           [TRACE]    | IDs: [(103568, 'ratingKey'), ... ] |
[2024-11-01 15:22:27,303] [builder.py:2235]           [DEBUG]    |                                                                                                    |
[2024-11-01 22:36:35,686] [overlays.py:587]           [INFO]     | 6333 Items found for Overlay File (0) Background                                                   |
[2024-11-01 22:36:35,708] [overlays.py:588]           [TRACE]    | Titles Found: ['*batteries not included', ... ] |

and then 8 hours to decide that most overlays don't need to change and apply may a dozen:

[2024-11-01 22:48:16,753] [overlays.py:73]            [INFO]     |====================================================================================================|
[2024-11-01 22:48:16,753] [overlays.py:73]            [INFO]     |                              Applying Overlays for the Movies Library                              |
...
[2024-11-02 06:50:46,153] [overlays.py:542]           [INFO]     |====================================================================================================|
[2024-11-02 06:50:46,154] [overlays.py:542]           [INFO]     |                                  Finished Movies Library Overlays                                  |

cold abyss Nov 2, 2024, 3:19 PM

#

The log shows it taking about 20 seconds per movie to decide if an overlay is needed or not

steady prism Nov 2, 2024, 3:19 PM

#

I added a few thousand movies to my test library and it's running right now

cold abyss Nov 2, 2024, 3:19 PM

#

You don't have anything running on the system that would limit how much CPU/Memory allocation is permitted?

#

overlay_files:   
- url: (redacted)

what is this url? The logs redact it, would be good to see what's in there.

You could try running that file locally rather than via a URL to see if it having to traverse the web is having an impact in some way?

steady prism Nov 2, 2024, 3:24 PM

#

It's that background overlay file.

cold abyss Nov 2, 2024, 3:24 PM

#

Gotcha, ty

steady prism Nov 2, 2024, 3:24 PM

#

Contents:

overlays:
## Creates a black background ribbon in the top left corner. Run this before any other top left overlay.
  Background:
    overlay:
      name: top-left-313-wide
      url: https://raw.githubusercontent.com/TheChrisK/PMM/main/overlays/background/top-left-313-wide.png #UPDATED TO URL FROM FILE CONFIG
      weight: 2
    plex_all: true

#

On the processing side there's nothing there. plex all, done.

cold abyss Nov 2, 2024, 3:26 PM

#

I can't see why it would be that, but could be one more thing to rule out

#

Keep everything local to the system and see if that has any impact

#

I doubt that's the root cause though

steady prism Nov 2, 2024, 3:27 PM

#

On my newly huge test library:

[2024-11-02 10:13:39,460] [plex.py:604]               [INFO]     | Loading All Movies from Library: Kometa-Demo-Movies                                                |
[2024-11-02 10:14:28,692] [plex.py:624]               [INFO]     | Loaded 8842 Movies                                                                                 |
[2024-11-02 10:14:28,716] [builder.py:2223]           [DEBUG]    |                                                                                                    |
[2024-11-02 10:14:28,717] [builder.py:2224]           [DEBUG]    | 8842 IDs Found                                                                                     |
[2024-11-02 10:14:28,723] [builder.py:2226]           [DEBUG]    |                                                                                                    |
[2024-11-02 10:21:12,708] [overlays.py:592]           [INFO]     | 8842 Items found for Overlay File (0) Background                                                   |

7 minutes to go from 8842 rating keys to 8842 movies vs his 7 hours for 6K

cold abyss Nov 2, 2024, 3:27 PM

#

I wonder if their CPU is being throttled by their motherboard or some other component on their system

#

They mentioned running an i7 7th gen which is about 7-8 years old now

#

It might be worth running some benchmarking tools on the system to see if something pops up

steady prism Nov 2, 2024, 3:28 PM

#

Sadly the trace log doesn't contain any more detail as to what going on:

[2024-11-02 10:14:28,717] [builder.py:2224]           [DEBUG]    | 8842 IDs Found
HERE
[2024-11-02 10:21:12,708] [overlays.py:592]           [INFO]     | 8842 Items found for Overlay File (0) Background

My test is running on an even lamer machine

cold abyss Nov 2, 2024, 3:29 PM

#

oh 😄

cold abyss Nov 2, 2024, 3:31 PM

#

steady prism Sadly the trace log doesn't contain any more detail as to what going on: ``` [20...

                    for method, value in builder.builders:
                        logger.debug("")
                        logger.debug(f"Builder: {method}: {value}")
                        logger.info("")
                        try:
                            builder.filter_and_save_items(builder.gather_ids(method, value))        # <<< THIS PRODUCES "8842 IDs Found"
                        except Failed as e:
                            if builder.ignore_blank_results:
                                logger.info("")
                                logger.warning(e)
                            else:
                                raise Failed(e)

                    added_titles = []
                    if builder.found_items:
                        for item in builder.found_items:
                            if builder.limit and len(added_titles) >= builder.limit:
                                break
                            key_to_item[item.ratingKey] = item
                            added_titles.append(item)
                            if item.ratingKey not in properties[prop_name].keys:
                                properties[prop_name].keys.append(item.ratingKey)
                    if added_titles:
                        logger.info(f"{len(added_titles)} Items found for {prop_name}")            # <<< THIS PRODUCES "8842 Items found for Overlay File (0) Background"

#

https://github.com/Kometa-Team/Kometa/blob/master/modules/overlays.py#L556

#

and this is the filter_and_save_items that is called:

https://github.com/Kometa-Team/Kometa/blob/master/modules/builder.py#L2219

steady prism Nov 2, 2024, 3:32 PM

#

#

Still actually applying overlays on mine, but in my case it's doing 2-3 per second

cold abyss Nov 2, 2024, 3:35 PM

#

I'd still be interested in seeing some benchmarks to show if there's some weird bottlenecking going on with OP's system

steady prism Nov 2, 2024, 3:35 PM

#

[2024-11-02 10:26:09,276] [overlays.py:201]           [INFO]     |
                                                                 | Birth
[2024-11-02 10:26:09,423] [overlays.py:524]           [INFO]     |   Overlays Applied: Overlay File (0) Background
[2024-11-02 10:26:09,456] [overlays.py:201]           [INFO]     |
                                                                 | Birth of the Dragon
[2024-11-02 10:26:09,457] [overlays.py:529]           [INFO]     |   Overlay Update Not Needed (Current Overlays: Overlay File (0) Background)
[2024-11-02 10:26:09,522] [overlays.py:201]           [INFO]     |
                                                                 | Birth/Rebirth
[2024-11-02 10:26:09,953] [overlays.py:524]           [INFO]     |   Overlays Applied: Overlay File (0) Background
[2024-11-02 10:26:10,031] [overlays.py:201]           [INFO]     |
                                                                 | Birthright Outlaw
[2024-11-02 10:26:10,696] [overlays.py:524]           [INFO]     |   Overlays Applied: Overlay File (0) Background
[2024-11-02 10:26:10,770] [overlays.py:201]           [INFO]     |
                                                                 | Bisbee '17
[2024-11-02 10:26:10,945] [overlays.py:524]           [INFO]     |   Overlays Applied: Overlay File (0) Background
[2024-11-02 10:26:11,025] [overlays.py:201]           [INFO]     |
                                                                 | The Bishop's Wife
[2024-11-02 10:26:11,218] [overlays.py:524]           [INFO]     |   Overlays Applied: Overlay File (0) Background

#

Mine is also created an asset directory for each, which I trimmed for space.

cold abyss Nov 2, 2024, 3:35 PM

#

Even if it's a newer CPU, maybe there's some incompatibility with the motherboard, or maybe the RAM has a slow clock speed, who knows - I'm not an expert at that stuff but throwing some shit and it might stick 😄

#

Although they did say it was fine one day on the system and not fine the next, maybe worth checking the Windows Update history to see if anything has been introduced, or maybe a driver, idk

steady prism Nov 2, 2024, 3:40 PM

#

Other facts for what they may be worth:

Mine:

[INFO]     |     Version: 2.1.0-build11 (Python 3.12.6) (Git: nightly)
[INFO]     |     Platform: Linux-6.8.0-45-generic-x86_64-with-glibc2.35
[INFO]     |     Total Memory: 15 GB
[INFO]     |     Available Memory: 8 GB

vs:

[INFO]     |     Version: 2.0.2 (Python 3.11.0) (Git: master)
[INFO]     |     Newest Version: 2.1.0
[INFO]     |     Platform: Windows-10-10.0.22631-SP0
[INFO]     |     Total Memory: 16 GB
[INFO]     |     Available Memory: 7 GB

Mine:

[INFO]     | Connected to server chaz-plex version 1.40.0.7998-c29d4c0c8
[INFO]     | Running on Linux version 6.8.0-45-generic
[INFO]     | Plex DB cache setting: 40 MB

vs:

[INFO]     | Connected to server minitopo version 1.41.1.9057-af5eaea7a
[INFO]     | Running on Windows version 10.0 (Build 22631)
[INFO]     | Plex DB cache setting: 4096 MB

tribal grotto Nov 2, 2024, 4:08 PM

#

sorry guys just getting back to this

#

so that was run locally from the Plex server

#

it's a Beelink i5-12450h with 16GB of DDR5 RAM

#

there's honestly not much else running on the box

cold abyss Nov 2, 2024, 4:08 PM

#

Is your Plex server running on Windows?

tribal grotto Nov 2, 2024, 4:08 PM

#

yes

cold abyss Nov 2, 2024, 4:09 PM

#

Gotcha

tribal grotto Nov 2, 2024, 4:09 PM

#

and this had never been an issue previously, as I posted that log from 10/15 where it only took 45 mins

#

then immediately the next day it was like 20 hrs

#

so for all intents and purposes it seemed to happen out of nowhere

cold abyss Nov 2, 2024, 4:10 PM

#

So the counter-argument to that is that nothing has changed on a Kometa side between Day 1 and Day 2 (I know we're not trying to place blame, just making the point)

tribal grotto Nov 2, 2024, 4:10 PM

#

yeah understood

cold abyss Nov 2, 2024, 4:10 PM

#

A lot can change on an operating system, but the Kometa environment itself did not change between those 2 runs

tribal grotto Nov 2, 2024, 4:10 PM

#

lemme see if there were any windows updates around that day

cold abyss Nov 2, 2024, 4:10 PM

#

Maybe check windows updates (idk if there's a way to check driver updates too) to see if something happened in that time

tribal grotto Nov 2, 2024, 4:11 PM

#

last updates were 10/10

#

drivers were prior to 10/15 as well

#

display drivers 10/10 and NIC drivers were back in July

cold abyss Nov 2, 2024, 4:12 PM

#

cold abyss ``` for method, value in builder.builders: ...

The period that went from 45m up to 7h, I don't believe any Plex calls are made during that time, instead it's all internal to Kometa and it's cache file, so I don't believe the network element is involved here.

I could be wrong about that, but based on this code I don't see any Plex calls happening

tribal grotto Nov 2, 2024, 4:13 PM

#

i also used the Plex built in SQL lite tool to check DB and it said it was fine

steady prism Nov 2, 2024, 4:14 PM

#

I suspect something in here:

                            key_to_item[item.ratingKey] = item

Since that's the only thing with any substance that happens in that 7-hour period

cold abyss Nov 2, 2024, 4:14 PM

#

So one suggestion, which you may not be comfortable with

#

Is to maybe pass your Plex URL and Token to Carrot, see if he can re-run his test against your library and see if there's any difference

#

If he's comfortable with that too of course

tribal grotto Nov 2, 2024, 4:15 PM

#

I guess if he's good with it I can do it

steady prism Nov 2, 2024, 4:15 PM

#

Assuming I can connect to it from here, sure.

tribal grotto Nov 2, 2024, 4:15 PM

#

yeah it should be open

cold abyss Nov 2, 2024, 4:16 PM

#

Maybe DM the details rather than in a public thread 😄

tribal grotto Nov 2, 2024, 4:16 PM

#

yes of course

#

one sec

cold abyss Nov 2, 2024, 4:17 PM

#

You should also update to 2.1.0 whenever you have a chance (and update the requirements as part of that process)

tribal grotto Nov 2, 2024, 4:17 PM

#

sent via PM

waxen barnBOT Nov 2, 2024, 4:18 PM

#

Moving between branches on a local system

NOTE: This is addressing changing from, for instance, latest to nightly.

replace BRANCH_NAME with latest, develop, or nightly; whichever you want to switch to.

cd DIRECTORY_WHERE_YOU_PUT_THE_CODE
git checkout BRANCH_NAME
git stash
git stash clear
git pull
# activate your venv here if you use one

python -m pip install -r requirements.txt

If you use a Virtual Environment [venv], make sure you are running inside your venv prior to running the python -m pip install -r requirements.txt command. Those stash commands are going to discard any changes you may have made to Kometa's own files. If that's a concern, you are probably familiar enough with git to know what you need to do instead.

cold abyss Nov 2, 2024, 4:18 PM

#

that will also work for updating 🙂

tribal grotto Nov 2, 2024, 4:19 PM

#

alright, i'll do this after Carrot tests

#

thanks for your help guys

steady prism Nov 2, 2024, 4:22 PM

#

First run against my library, which actually applied the background overlay to all 8K movies, took an hour and ten.

tribal grotto Nov 2, 2024, 4:23 PM

#

man so weird

steady prism Nov 2, 2024, 4:29 PM

#

Goes through all the "Parsing 123/456", then sits here for a while:

#

not 7 hours, but a while

tribal grotto Nov 2, 2024, 4:35 PM

#

And no idea what it’s doing at that stage?

steady prism Nov 2, 2024, 4:35 PM

#

second run where it only had to report that 8K overlays didn't need updating, 13 minutes.

tribal grotto Nov 2, 2024, 4:35 PM

#

Wow

#

What the heck

steady prism Nov 2, 2024, 4:36 PM

#

OK, started run against minitopo

tribal grotto Nov 2, 2024, 4:37 PM

#

This will be interesting

steady prism Nov 2, 2024, 4:37 PM

#

I commented out all collections and the TV library just to make sure it was only overlays.

tribal grotto Nov 2, 2024, 4:38 PM

#

What makes that different than just running overlays-only

steady prism Nov 2, 2024, 6:09 PM

#

Nothing really; just making sure I don't touch anything else.

tribal grotto Nov 2, 2024, 6:33 PM

#

Got it

#

Wait, I just thought of something that seems trivial but maybe it’s not?

Historically all my movies were:

Z:\movie_folder\movie_file

I did a bunch of rips with better audio and video, and many of those are just now in the root of Movies, like

Z:\movie_name

this wouldn’t be slowing down Kometa would it?

zealous blaze Nov 2, 2024, 6:47 PM

#

Kometa doesn't care where files are stored, it uses Plex library

tribal grotto Nov 2, 2024, 6:56 PM

#

Fair enough jjust throwing it out there

tribal grotto Nov 2, 2024, 8:04 PM

#

Is it possible to have metadata corruption?

cold abyss Nov 2, 2024, 8:06 PM

#

DB Repair should deal with a lot of the common things like that I think

steady prism Nov 2, 2024, 8:26 PM

#

OK, back after some errands. Still running.

#

Time to process the background overlay:

[2024-11-02 11:39:49,626] [plex.py:624]               [INFO]     | Loaded 6336 Movies                                                                                 |
[2024-11-02 11:39:49,644] [builder.py:2223]           [DEBUG]    |                                                                                                    |
[2024-11-02 11:39:49,644] [builder.py:2224]           [DEBUG]    | 6336 IDs Found                                                                                     |
[2024-11-02 11:39:49,649] [builder.py:2226]           [DEBUG]    |                                                                                                    |
[2024-11-02 11:47:29,999] [overlays.py:592]           [INFO]     | 6336 Items found for Overlay File (0) Background                                                   |

About 8 minutes, not too bad. In line with my 8K.

tribal grotto Nov 2, 2024, 8:28 PM

#

Wait so it’s all done?

steady prism Nov 2, 2024, 8:29 PM

#

No, still running.

tribal grotto Nov 2, 2024, 8:29 PM

#

Oh oh

#

Ok sorry was confused

#

So the background overlay is on par with your library test

#

But it’s still gotta do resolution and audio

steady prism Nov 2, 2024, 8:31 PM

#

All overlays processed [figuring out what overlays go on what items] in under 30 minutes:

[2024-11-02 11:38:56,546] [overlays.py:23]            [INFO]     |                                      Movies Library Overlays
[2024-11-02 11:38:56,546] [overlays.py:23]            [INFO]     |=============================================================

[2024-11-02 12:06:49,873] [overlays.py:610]           [INFO]     |=============================================================
[2024-11-02 12:06:49,873] [overlays.py:610]           [INFO]     |                              Overlay Operation for the Movies Library

cold abyss Nov 2, 2024, 8:32 PM

#

~~So that rules out your Plex setup itself being the issue~~

tribal grotto Nov 2, 2024, 8:33 PM

#

I guess that’s comforting

steady prism Nov 2, 2024, 8:33 PM

#

Applying overlays has been running for 2.5 hours and has only gotten to the Cs

[2024-11-02 12:07:45,903] [overlays.py:73]            [INFO]     |                              Applying Overlays for the Movies Library
...
                                                                 | Captain Corelli's Mandolin
[2024-11-02 15:25:50,269] [overlays.py:524]           [INFO]     |   Overlays Applied: Overlay File (0) Background, Overlay File (1) 720P, Overlay File (2) AAC

cold abyss Nov 2, 2024, 8:33 PM

#

Oh

tribal grotto Nov 2, 2024, 8:33 PM

#

Oh interesting

#

So what would make that so sluggish?

cold abyss Nov 2, 2024, 8:33 PM

#

So maybe something like the write speeds to the disk?

#

Or again it could be Plex being sluggish with the uploads

steady prism Nov 2, 2024, 8:35 PM

#

Applying the overlay is:

Hey Plex give me the art for this thing
create local overlaid art
Hey Plex here's a new image for you to use
Thanks for confirming, Plex.

Step 2 is common with my library [same code, disk, etc], so probably not involved

cold abyss Nov 2, 2024, 8:35 PM

#

I don't know if there's also:

4. Plex confirms "Yup I've done that"

#

You would maybe need a -lr log to see how long each communication is taking?

tribal grotto Nov 2, 2024, 8:37 PM

#

But if I’m running kometa on the actual plex server what could cause such long delays?

cold abyss Nov 2, 2024, 8:37 PM

#

for topo's reference, turning on -lr logs every network connection from/to Kometa, it's the ultimate verbose logging

#

Plex could be slow to say "here's the art"
Downloading the base image could be slow
Ceating the local overlaid art could be slow (unlikely given it's common as Carrot said)
Uploading the new artwork could be slow
Applying the Overlay label could go slow
Plex could take a while to respond with "yup I've set that artwork"

tribal grotto Nov 2, 2024, 8:39 PM

#

Is there a setting or something in Plex that would affect that? Like I said it was all clockwork for many months

#

And one day it was fine and the next not

cold abyss Nov 2, 2024, 8:39 PM

#

This isn't specific to the application of overlays though, given that your log took 7 hours just to process the 6K IDs compared to Carrot's which was minutes rather than hours

cold abyss Nov 2, 2024, 8:39 PM

#

tribal grotto Is there a setting or something in Plex that would affect that? Like I said it w...

When did you last update Plex?

tribal grotto Nov 2, 2024, 8:40 PM

#

Great question lemme check

#

running Version 1.41.1.9057 which says current

#

lemme see when that was released

cold abyss Nov 2, 2024, 8:41 PM

#

released vs you updating to it isn't necessarily the same thing

tribal grotto Nov 2, 2024, 8:41 PM

#

true

#

that version has been out since September 27, 2024

#

maybe i'll check the exe for a modified date?

steady prism Nov 2, 2024, 8:43 PM

#

I'm going to cancel this run given that we have the information we need from it.

cold abyss Nov 2, 2024, 8:45 PM

#

try uploading this image to Plex as a poster, see how long it takes?

star-milky-way-cosmos-atmosphere-panorama-telescope-734956-pxhere.com.jpg

tribal grotto Nov 2, 2024, 8:45 PM

#

10/24 was update

cold abyss Nov 2, 2024, 8:45 PM

#

It's 8.3mb so approaching the upper limit of Plex's max size (10mb)

tribal grotto Nov 2, 2024, 8:45 PM

#

just to any random movie or show?

cold abyss Nov 2, 2024, 8:46 PM

#

do it to maybe 5 movies

#

the log was a bit inconsistent that some would take 20 seconds to process and some would be pretty instant

tribal grotto Nov 2, 2024, 8:47 PM

#

one sec i'm using discord via web browser and it doesn't appear that i can download that

cold abyss Nov 2, 2024, 8:47 PM

#

lemme upload it

#

https://i.ibb.co/WkV8yPN/star-milky-way-cosmos-atmosphere-panorama-telescope-734956-pxhere-com.jpg

tribal grotto Nov 2, 2024, 8:48 PM

#

ok thanks

#

i mean it adds it to the movie instantly

cold abyss Nov 2, 2024, 8:49 PM

#

Try it on a few

tribal grotto Nov 2, 2024, 8:50 PM

#

one sec kid calling me

cold abyss Nov 2, 2024, 8:50 PM

#

Next step - create a "test" movie library with lets say 10-20 movies, see if your overlay application is slow on that library

tribal grotto Nov 2, 2024, 8:53 PM

#

alright sorry

#

i added it to 5 movies and it applied instantly each time

#

zero delay

#

alright i actually have a library that exists with nothing in it, i could easily copy a dozen movies over and target it

#

running

#

appears to be flying

#

it's parsing collections but visually going much quicker

#

created 3 collections thus far

#

so weird, just parsing the movies for the collections is insanely fast by comparison

#

i suppose i should've skipped collections but it's interesting to see how fast it's working. already built 9 collections

#

or maybe it's not interesting since this isn't the issue 🙂

#

started overlays

#

it completed

#

total script runtime 11 mins

#

all overlays applied

steady prism Nov 2, 2024, 9:14 PM

#

Now on a second run against minitopo the "translate from these 6336 IDs to movies" is far slower

#

20 seconds each or so.

tribal grotto Nov 2, 2024, 9:17 PM

#

that's sort of odd right?

steady prism Nov 2, 2024, 9:19 PM

#

Well, it's in line with what you see.

tribal grotto Nov 2, 2024, 9:19 PM

#

so good but bad 🙂

#

i just noticed something odd, under scheduled tasks in Plex, everything was checked EXCEPT update all libraries during maintenance

#

could this somehow be causing an issue?

cold abyss Nov 2, 2024, 10:34 PM

#

No I don't think so

cold abyss Nov 2, 2024, 10:35 PM

#

tribal grotto total script runtime 11 mins

Now do a trace run log and we can see how long each movie to say overlay update not needed

tribal grotto Nov 2, 2024, 10:57 PM

#

alright

#

i mean

#

3:38

cold abyss Nov 2, 2024, 11:01 PM

#

Post the log?

tribal grotto Nov 2, 2024, 11:01 PM

#

yeah one sec copying it over

#

📎 meta.log

lament impBOT Nov 2, 2024, 11:02 PM

#

📝 If you want to review this again, topochocolate:
:one: Right-click (or long press with phone) on the message that contains the log
:two: Select: Copy Message Link
:three: Use the command: /logscan <message_link> or !logscan <message_link> and paste the value copied from the previous step where you see <message_link> 📝

#

**User Info**

Author of Linked Message: topochocolate
Person who Invoked the Command: topochocolate
File Name: meta.log

Table of Contents:
Page 01: User Info
Page 02: Kometa Info
Page 03: Kometa Summary Info
Page 04: Kometa Config.yml YAML Validation ✅
Page 05: Plex Configuration - Section 1
Page 06: Rec 01 - ❌ ANIDB AUTH ERRORS
Page 07: Rec 02 - ❌🔒 BLANK API KEY ERROR
Page 08: Rec 03 - ❌ [ERROR]
Page 09: Rec 04 - ❌ MY ANIME LIST CONNECTION ERROR
Page 10: Rec 05 - ❌⏰ KOMETA SCHEDULED TIME CONFLICT
Page 11: Rec 06 - ❌ TRAKT CONNECTION ERROR
Page 12: Rec 07 - ⚠️ [WARNING]
Page 13: Rec 08 - ⚠️ NO ITEMS FOUND IN PLEX
Page 14: Rec 09 - ⚠️ PLEX REGEX ERROR
Page 15: Rec 10 - 💬 CONVERT WARNING

#

📎 parsed_meta.log_config_topochocolate.yml

cold abyss Nov 2, 2024, 11:05 PM

#

That looks much better

#

They are all processed within 1 second basically

tribal grotto Nov 2, 2024, 11:05 PM

#

right

cold abyss Nov 2, 2024, 11:05 PM

#

So maybe there's some corruption happening/happened in the Movies library that DB Repair can't detect/fix

#

You could try a duplicate library "Movies 2" and if that works, then delete "Movies" and rename "Movies 2" back to "Movies" ?

tribal grotto Nov 2, 2024, 11:08 PM

#

hmmmmm

#

interesting thought

#

gimme a bit gotta make my kid dinner

cold abyss Nov 2, 2024, 11:09 PM

#

Carrot has a script that can backup your watch history and then reapply it to the new library if needed

steady prism Nov 2, 2024, 11:14 PM

#

If both libraries exist at the same time that should transfer automatically.

tribal grotto Nov 2, 2024, 11:19 PM

#

what about external users?

cold abyss Nov 2, 2024, 11:20 PM

#

steady prism If both libraries exist at the same time that should transfer automatically.

I'm not 100% convinced on that as the ID for the library will be different

tribal grotto Nov 2, 2024, 11:20 PM

#

ah damn

cold abyss Nov 2, 2024, 11:20 PM

#

I assume Plex works on a unique ID for the library rather than the library name itself?

cold abyss Nov 2, 2024, 11:20 PM

#

tribal grotto what about external users?

The script also handles this (if it's needed)

#

I've used it on more than one occasion when moving servers

tribal grotto Nov 2, 2024, 11:20 PM

#

hmmmmm

cold abyss Nov 2, 2024, 11:22 PM

#

The only hiccup I had is that my On Deck stuff didn't show up, I don't think the script does "in progress" episodes/movies

#

It's been a long time so I may be misremembering that

tribal grotto Nov 2, 2024, 11:22 PM

#

would i also have to reset all the posters?

cold abyss Nov 2, 2024, 11:22 PM

#

When you say reset?

tribal grotto Nov 2, 2024, 11:23 PM

#

i hate most of the default ones that have too much writing

cold abyss Nov 2, 2024, 11:23 PM

#

It would be a fresh library, so it would be "clean" of any changes that were made to the previous library

tribal grotto Nov 2, 2024, 11:23 PM

#

i've probably changed like 85% of all movie poster to something with just the title as oppose to the Plex agent default

#

oh god

cold abyss Nov 2, 2024, 11:23 PM

#

Carrot has another script that can grab current artwork from your library

#

which you could use as the asset directory for the new library

tribal grotto Nov 2, 2024, 11:24 PM

#

ok so it 100% appears to be something with Movies

#

i just ran overlays-only against TV and it only took 15 mins

cold abyss Nov 2, 2024, 11:24 PM

#

Have you tried running ImageMaid before?

tribal grotto Nov 2, 2024, 11:25 PM

#

negative

cold abyss Nov 2, 2024, 11:25 PM

#

Maybe if you have like a LOT of historical overlay images, if the metadata folder is bloated to heck it could slow things down?

waxen barnBOT Nov 2, 2024, 11:25 PM

#

Worried about Image Bloat?

Use ImageMaid to get rid of unwanted bloat!

Features include:
:one: Cleaning up of custom posters and title cards that were uploaded to Plex and are not in use anymore
:two: Deleting the Phototranscoder folder to free up even more space
:three: Perform the Empty Trash, Clean Bundles and Optimize DB operations
:four: Scheduling ImageMaid to run regularly

cold abyss Nov 2, 2024, 11:26 PM

#

when a new poster is uploaded, Plex and Kometa don't remove the old one(s).

i.e. if you use ratings on the poster and one of the ratings change, then a new poster is uploaded each time that rating changes

tribal grotto Nov 2, 2024, 11:26 PM

#

so #3 i do have those checked to run weekly in Plex

cold abyss Nov 2, 2024, 11:26 PM

#

That's fine, it's just one of the things ImageMaid can do too

tribal grotto Nov 2, 2024, 11:27 PM

#

yeah I could try and run this

cold abyss Nov 2, 2024, 11:27 PM

#

You could look at the posters for a couple of your movies in Plex and see if there's a boatload of overlay'd ones in there

#

Example here, the red border ones are inactive ones that are just bloating the system

#

The green is the current one

tribal grotto Nov 2, 2024, 11:28 PM

#

yea there's 6 on one of the first movies i clicked on

cold abyss Nov 2, 2024, 11:28 PM

#

Three inactive isn't particularly excessive, but each image is using space

tribal grotto Nov 2, 2024, 11:28 PM

#

i'm guessing that's pretty standard as this movie has been in my library since nearly day 1

cold abyss Nov 2, 2024, 11:28 PM

#

6 doesn't seem excessive either

tribal grotto Nov 2, 2024, 11:29 PM

#

well, times 6000

cold abyss Nov 2, 2024, 11:29 PM

#

It's maybe another tool for the belt to keep things clean 😄

#

you're welcome to try it though, if it makes a difference then great

tribal grotto Nov 2, 2024, 11:29 PM

#

how long is likely going to take to run, and will it hamper Plex while it's running?

cold abyss Nov 2, 2024, 11:29 PM

#

Plex needs to be offline whilst it runs

tribal grotto Nov 2, 2024, 11:29 PM

#

ah damn

cold abyss Nov 2, 2024, 11:30 PM

#

It doesn't take long

#

Minutes as opposed to hours

tribal grotto Nov 2, 2024, 11:30 PM

#

alright, i might run it tomorrow

#

or maybe monday

cold abyss Nov 2, 2024, 11:30 PM

#

👍

tribal grotto Nov 2, 2024, 11:30 PM

#

thanks for the suggestion

cold abyss Nov 2, 2024, 11:30 PM

#

You could set up your second library in the meantime so that all of it it scanned in and ready if imagemaid doesn't improve things

#

If you're thinking about backing up the original library's posters to use on a new library, Carrot would need to confirm if you first need to run remove_overlays: true so that the posters are clean, or if it knows to fetch the original poster that Kometa backed up.

#

Just thinking that removing the overlays may take as much time or more than the application of them, so that may be a lengthy task to take too 😄

tribal grotto Nov 2, 2024, 11:32 PM

#

ha ha

#

kill me

#

i might start running Kometa from my work machine lol

#

core i9 with 64GBs of RAM on a big ten university network with loads of bandwidth

#

just let that thing run for 20 hrs

cold abyss Nov 2, 2024, 11:34 PM

#

hahaha

steady prism Nov 2, 2024, 11:44 PM

#

cold abyss I assume Plex works on a unique ID for the library rather than the library name ...

I think it links by title. As I create new test libraries on this server they reflect the same watched status without me doing anything.

tribal grotto Nov 2, 2024, 11:46 PM

#

semi-comforting

steady prism Nov 2, 2024, 11:58 PM

#

If all your movies have overlays I have another script that will turn those backup images into an asset directory.

tribal grotto Nov 2, 2024, 11:58 PM

#

what exactly does that do?

steady prism Nov 2, 2024, 11:59 PM

#

It renames the backup images into a folder structure ready for Kometa's assets.

tribal grotto Nov 2, 2024, 11:59 PM

#

does that help with performance?

#

or is it just redundancy?

cold abyss Nov 3, 2024, 12:00 AM

#

It's to help you move your posters from your old library to your new one

#

Kometa backs up the original image before it applies an Overlay to it

#

So we could use that as the "source of truth" for images on the new library, rather than you having to edit all your posters manually

tribal grotto Nov 3, 2024, 12:01 AM

#

got it

steady prism Nov 3, 2024, 3:01 PM

#

I am just running a test this morning and now the parsing is flying again.

No changes here so it's something on that Plex server or in that library.

tribal grotto Nov 3, 2024, 7:15 PM

#

Interesting

#

So I did a refresh all metadata on Movies, and checked the box to update library during maintenance

#

Maint obviously ran last night

#

I wonder if there was metadata buildup or something taking place

#

I haven’t looked at it much today my kid had swim this morning and I’ve got a buddy coming by for football in a few

tribal grotto Nov 4, 2024, 3:07 PM

#

i ran image maid against the server early this morning

#

oddly my parsing is back to crawling

#

oops, I'm an idiot, I ran it in report mode

tribal grotto Nov 4, 2024, 3:49 PM

#

cold abyss Nov 4, 2024, 4:18 PM

#

tribal grotto oops, I'm an idiot, I ran it in report mode

That is normally what we suggest so that you have an idea what it's gonna do 😄

#

Report
Move
do manual checks to make sure everything looks ok
Delete

tribal grotto Nov 4, 2024, 4:18 PM

#

yeah i moved them and confirmed all looks good

#

however rerunning overlays only and it's still crawling

#

toggled my VPN and windows firewall ro rule those out

cold abyss Nov 4, 2024, 4:19 PM

#

Do you have the ability to maybe try running Kometa in Docker to see if that behaves any differently?

tribal grotto Nov 4, 2024, 4:19 PM

#

i don't have docker setup

cold abyss Nov 4, 2024, 4:19 PM

#

Actually Carrot saw the slowness too, so I doubt that's gonna help

#

Maybe time to try a 2nd library?

tribal grotto Nov 4, 2024, 4:20 PM

#

i realize that's probably the only answer

#

but i really don't want to go through that

cold abyss Nov 4, 2024, 4:20 PM

#

I get it

tribal grotto Nov 4, 2024, 4:21 PM

#

especially when Plex itself is behaving perfectly fine for me and my users

#

this might be a dumb Q, but would it go any quicker if the overlay files were saved locally? like instead of this call:

| Validating Method: overlay |
| Value: {'name': 'top-left-313-wide', 'url': 'https://raw.githubusercontent.com/TheChrisK/PMM/main/overlays/background/top-left-313-wide.png', 'weight': 2} |

#

| Validating Method: overlay

#

it's validating based on c:\whatever\overlay.png

cold abyss Nov 4, 2024, 4:22 PM

#

I did propose this earlier to see if there's any difference

tribal grotto Nov 4, 2024, 4:22 PM

#

so just copy the movie overlays down locally and edit the config to call them there?

cold abyss Nov 4, 2024, 4:22 PM

#

Probably got lost in all the other stuff 😄

#

Yeah you can give it a go

tribal grotto Nov 4, 2024, 4:23 PM

#

alright maybe i'll try that

#

thanks

cold abyss Nov 4, 2024, 4:23 PM

#

That's nothing to do with Parsing though, FYI

#

If your parsing is slow, there's no correlation between that and the overlay itself

tribal grotto Nov 4, 2024, 4:23 PM

#

oh? i assumed each parse was comparing to that URL

#

i see

cold abyss Nov 4, 2024, 4:24 PM

#

Parsing is fetching the IDs of the items in the library

#

the parsing is based on the builder (in this case, plex_all) rather than the overlay image

tribal grotto Nov 4, 2024, 4:25 PM

#

damn

cold abyss Nov 4, 2024, 4:25 PM

#

You could maybe pull the Plex logs to see if there's anything happening in there

#

Normally if Plex is doing its own thing, it doesn't like responding to requests from Kometa quickly 😄

#

It's why we don't suggest running Kometa during Plex's scheduled maintenance window

tribal grotto Nov 4, 2024, 4:26 PM

#

which Plex log what i want to look at?

cold abyss Nov 4, 2024, 4:27 PM

#

Good question

#

I don't really know much about the logs themself

#

Carrot may know more about that

tribal grotto Nov 4, 2024, 4:29 PM

#

right on

steady prism Nov 4, 2024, 4:30 PM

#

I think there's just the one of interest; the media server log.

tribal grotto Nov 4, 2024, 4:31 PM

#

got it

#

dunno how normal this is, but it's filled with this stuff:

Nov 04, 2024 09:11:49.826 [68616] DEBUG - Audio Stream: 545676, Subtitle Stream: 545677
Nov 04, 2024 09:11:49.827 [68616] DEBUG - We're going to try to auto-select an audio stream for account 1.
Nov 04, 2024 09:11:49.827 [68616] DEBUG - Selecting best audio stream for part ID 219322 (language: en)
Nov 04, 2024 09:11:49.827 [68616] DEBUG - We're going to try to auto-select a subtitle for account 1.
Nov 04, 2024 09:11:49.827 [68616] DEBUG - Subtitles: Found a candidate subtitle language [en] for a foreign film
Nov 04, 2024 09:11:49.827 [68616] DEBUG - Audio Stream: 545570, Subtitle Stream: 545572
Nov 04, 2024 09:11:49.828 [68616] DEBUG - We're going to try to auto-select an audio stream for account 1.
Nov 04, 2024 09:11:49.828 [68616] DEBUG - Selecting best audio stream for part ID 219321 (language: en)
Nov 04, 2024 09:11:49.828 [68616] DEBUG - We're going to try to auto-select a subtitle for account 1.
Nov 04, 2024 09:11:49.828 [68616] DEBUG - Subtitles: Found a candidate subtitle language [en] for a foreign film
Nov 04, 2024 09:11:49.828 [68616] DEBUG - Audio Stream: 545567, Subtitle Stream: 545568
Nov 04, 2024 09:11:49.830 [68616] DEBUG - We're going to try to auto-select an audio stream for account 1.
Nov 04, 2024 09:11:49.830 [68616] DEBUG - Selecting best audio stream for part ID 219320 (language: en)
Nov 04, 2024 09:11:49.830 [68616] DEBUG - We're going to try to auto-select a subtitle for account 1.
Nov 04, 2024 09:11:49.830 [68616] DEBUG - Subtitles: Found a candidate subtitle language [en] for a foreign film
Nov 04, 2024 09:11:49.830 [68616] DEBUG - Audio Stream: 545564, Subtitle Stream: 545565
Nov 04, 2024 09:11:49.831 [68616] DEBUG - We're going to try to auto-select an audio stream for account 1.
Nov 04, 2024 09:11:49.831 [68616] DEBUG - Selecting best audio stream for part ID 219319 (language: en)

steady prism Nov 4, 2024, 4:35 PM

#

Seems like other things are going on. Is something playing?

tribal grotto Nov 4, 2024, 4:35 PM

#

no

#

nobody streaming

steady prism Nov 4, 2024, 4:36 PM

#

This just looks like something related to playing a stream:

Nov 04, 2024 09:43:28.631 [62640] DEBUG - We're going to try to auto-select an audio stream for account 1.
Nov 04, 2024 09:43:28.631 [62640] DEBUG - Selecting best audio stream for part ID 219318 (language: en)
Nov 04, 2024 09:43:28.631 [62640] DEBUG - We're going to try to auto-select a subtitle for account 1.
Nov 04, 2024 09:43:28.631 [62640] DEBUG - Subtitles: Found a candidate subtitle language [en] for a foreign film
Nov 04, 2024 09:43:28.631 [62640] DEBUG - Audio Stream: 545558, Subtitle Stream: 545559

tribal grotto Nov 4, 2024, 4:37 PM

#

ok wait this is weird

#

the #### that's my work desktop

#

Plex wasn't playing anything but the App was open

#

i just closed it

#

lemme try again

#

it was just sitting at the Plex home screen i don't know why it would be doing anything

#

well it's not exactly flying

#

but it appears to be moving slightly quicker

#

📎 message.txt

#

so it appears to be doing it less freqently

#

why is it listing subtitle?

#

is that just the general parsing?

steady prism Nov 4, 2024, 4:53 PM

#

~~I don't think this parsing in Kometa hits Plex at all.~~ turns out it does

tribal grotto Nov 4, 2024, 5:02 PM

#

huh

#

wtf

#

wait i gotta figure out what this 192.168 IP is

steady prism Nov 4, 2024, 5:04 PM

#

That's just a guess, but it's taking a list of Rating Keys which it just got from Plex and converting those to movies, which as far as I know comes from the Kometa cache.

~~I'm pretty sure it's not hitting Plex for everything in that list.~~ In fact it is

tribal grotto Nov 4, 2024, 5:05 PM

#

ok that IP is def this desktop i'm running Kometa from

#

and Plex is closed

#

verified not open in any browsers

steady prism Nov 4, 2024, 5:10 PM

#

Oh, looks like it does:

| http://192.168.1.11:32400 "GET /library/metadata/80629 HTTP/11" 200 2440                           |
| http://192.168.1.11:32400 "GET /library/metadata/98240 HTTP/11" 200 2476                           |
| http://192.168.1.11:32400 "GET /library/metadata/98227 HTTP/11" 200 5961                           |
Parsing 1234/8842

tribal grotto Nov 4, 2024, 5:14 PM

#

well i understand why it's asking for the audio stream

#

because that's part of the overlay

#

but why would it need to grab subs too? i feel like that's an extra 6300 unnecessary requests

#

again unless that's just part of the default audio get

steady prism Nov 4, 2024, 5:23 PM

#

Kometa is making a single request for information on the thing; it's not making separate requests for individual components.

#

What Plex does internally is not under Kometa control.

#

| http://192.168.1.11:32400 "GET /library/metadata/80629 HTTP/11" 200 2440                           |
| http://192.168.1.11:32400 "GET /library/metadata/98240 HTTP/11" 200 2476                           |
| http://192.168.1.11:32400 "GET /library/metadata/98227 HTTP/11" 200 5961                           |

One request to Plex from Kometa for each rating key

tribal grotto Nov 4, 2024, 5:27 PM

#

i see

#

so is having subs technically slowing it down?

steady prism Nov 4, 2024, 5:32 PM

#

No.

tribal grotto Nov 4, 2024, 5:34 PM

#

ok that's comforting

#

so roughly an hour in and it's parsed 600 titles

#

which is way faster than this morning

#

so unsure why but Plex being open on my work desktop not doing anything was def slowing it down

#

ok just curious about this:

📎 message.txt

#

Plex seems to be updating available extras for a movie in the library?

#

Carrot you testing Kometa on me? I see some WAN reqs

steady prism Nov 4, 2024, 5:44 PM

#

tribal grotto Carrot you testing Kometa on me? I see some WAN reqs

I just made a couple small requests. Not running Kometa.

#

Writing up details now.

tribal grotto Nov 4, 2024, 5:44 PM

#

oh ok

#

no worries

steady prism Nov 4, 2024, 5:48 PM

#

Here are some comparison numbers to illustrate the core of the issue:

I'm running curl requests outside Kometa on this Celeron machine in my home.

First, connecting to the plex server that is runnign on this same machine:

Retrieve a list of everything in the library

> time curl -fLv "http://192.168.1.11:32400/library/sections/1/all?includeGuids=1&type=1&X-Plex-Token=TEST_TOKEN"
real    7.41s

an XML file containing all details on the 8844 items in that library came back in 7.41 seconds.

Now let's ask Plex for the details on one item [this is the request that Plex does for everything during the Parsing 1234/5678:

time curl -fLv "http://192.168.1.11:32400/library/metadata/73535?X-Plex-Token=TEST_TOKEN"
real    0.01s

1/100th of a second.

Now, that server is on the same machine, so let's try my production Plex server across the country:

time curl -fLv "https://prod_plex/library/sections/19/all?includeGuids=1&type=1&X-Plex-Token=PROD_TOKEN"
real    8.01s

This library has 8538 items; info on all of them retrieved in 8 seconds, pretty comparable to the local server.

Now let's get info on the last item in the list:

time curl -fLv "https://prod_plex/library/metadata/1035113?X-Plex-Token=PROD_TOKEN"
real    0.23s

Took longer, a quarter of a second, which is not entirely surprising since the server is 2000 miles away.

Now let's try against yours.

Get the list of everything in the library:

time curl -fLv "http://topochocolate/library/sections/1/all?includeGuids=1&type=1&X-Plex-Token=TOPO_TOKEN"
real    15.11s

Twice as long as either of mine.

And now the request for information about an individual thing, again the request that Kometa will make for every item in the library during Parsing :

time curl -fLv "http://topochocolate/library/metadata/125610?X-Plex-Token=TOPO_TOKEN"
real    24.49s

Almost 30 seconds per item.

No idea why your server is behaving this way, but that's the specific reason for the slow parsing. Plex is just slow responding. and there's nothing Kometa can do about that.

#

I would not be surprised to find that the same thing is happening with the requests that happen during overlay application.

tribal grotto Nov 4, 2024, 5:50 PM

#

this is so frustrating

steady prism Nov 4, 2024, 5:51 PM

#

One other difference I suppose is that both my servers are running under ubuntu, so there may be a Plex-on-Windows component here as well.

tribal grotto Nov 4, 2024, 5:52 PM

#

i mean, that would make more sense if Plex and/or Windows had a major update after the 15th

#

the fact that on the 15th it was quick

#

and then the 16th is just got insanely slow

#

is the part that's killing me

tribal grotto Nov 4, 2024, 6:09 PM

#

gotta step away for a few hours got some meetings and what not

steady prism Nov 4, 2024, 6:36 PM

#

Plex logs generated while responding to one of those individual item requests:

Mine:

Nov 04, 2024 12:27:57.833 [123415730072376] DEBUG - Request: [192.168.1.11:35250 (Subnet)] GET /library/metadata/93205 (4 live) #199fe2 GZIP Signed-in Token (chazlarson) (random)
Nov 04, 2024 12:27:57.838 [123415730072376] DEBUG - We're going to try to auto-select an audio stream for account 1.
Nov 04, 2024 12:27:57.838 [123415730072376] DEBUG - Selecting best audio stream for part ID 201978 (language: en)
Nov 04, 2024 12:27:57.838 [123415730072376] DEBUG - We're going to try to auto-select a subtitle.
Nov 04, 2024 12:27:57.838 [123415730072376] DEBUG - Subtitles: Found a candidate subtitle language [en] for a foreign film
Nov 04, 2024 12:27:57.838 [123415730072376] DEBUG - Audio Stream: 673656, Subtitle Stream: 673657
Nov 04, 2024 12:27:57.841 [123416845757240] DEBUG - Completed: [192.168.1.11:35250] 200 GET /library/metadata/93205 (4 live) #199fe2 GZIP 7ms 4065 bytes (pipelined: 2272)

elapsed 841 - 833 = 8 ms

Yours [from your log snippet above]:

Nov 04, 2024 10:41:25.996 [74740]           DEBUG - Request: [192.168.1.79:62523 (Allowed Network (Subnet))] GET /library/metadata/79640 (3 live) #284d GZIP Signed-in Token (mtopo) (minitopo)
Nov 04, 2024 10:41:26.014 [74740]           DEBUG - We're going to try to auto-select an audio stream for account 1.
Nov 04, 2024 10:41:26.014 [74740]           DEBUG - Selecting best audio stream for part ID 115071 (language: en)
Nov 04, 2024 10:41:26.014 [74740]           DEBUG - We're going to try to auto-select a subtitle for account 1.
Nov 04, 2024 10:41:26.014 [74740]           DEBUG - Subtitles: Found a candidate subtitle language [en] for a foreign film
Nov 04, 2024 10:41:26.014 [74740]           DEBUG - Audio Stream: 283244, Subtitle Stream: 517258
Nov 04, 2024 10:41:26.017 [70096]           DEBUG - Completed: [192.168.1.79:62523] 200 GET /library/metadata/79640 (3 live) #284d GZIP 21ms 4162 bytes (pipelined: 166)

elapsed 1017 - 996 = 21 ms

Effectively all the time difference in that first line

Perhaps higher log level in Plex would show something?

cold abyss Nov 4, 2024, 7:28 PM

#

steady prism Here are some comparison numbers to illustrate the core of the issue: I'm runni...

It might be interesting to know if this is specific to the "Movies" library or if other libraries behave in the same manner

#

Would give an idea as to if creating a new Movies library would be of any benefit

steady prism Nov 4, 2024, 7:45 PM

#

There are two other libraries on that server; "TV Shows", which is empty, and "Anime", which contains 16 items. Neither of them take longer than .12s to respond to the "all-items" request, while the Movies library with 6337 items typically takes 15 seconds, but sometimes 5.

A couple minutes ago, five requests in a row took 15 seconds each; now two requests have taken 5 seconds each.

#

A request for an individual item in the Anime library is taking a few hundredths of a second.

tribal grotto Nov 4, 2024, 8:22 PM

#

ok i may have fixed it

#

yes Anima was my test library

#

it only had one thing for my wife

#

*Anime

#

so i kept thinking about that, what's different between these libraries

#

and the only thing I came back to (aside from general amount of titles) was the fact that a good amount of movies were just in the root of Movies and not in their own folder as I upgraded their picture/audio

#

i found this thread:

https://www.reddit.com/r/PleX/comments/10wii1c/movies_in_one_big_folder_vs_individual_folders/

From the PleX community on Reddit

Explore this post and more from the PleX community

#

used the script to create folders for all those loose movie files in Movies

#

rescanned the library

#

and i've now parsed 1130 titles in about 14 minutes

tribal grotto Nov 4, 2024, 9:52 PM

#

it's applying overlays now, so it looks like it might finish in the next 10 mins or so

#

total runtime will be maybe an hour and 50 mins

tribal grotto Nov 5, 2024, 12:34 AM

#

correction, 2:42:02 with TV overlays too

tribal grotto Nov 5, 2024, 9:21 PM

#

@steady prism @cold abyss I'm going to close this out, I'm satisified, most recent run was only 45 mins. Thank you for all your help!

#Kometa taking 20+ hours to complete