✅ YAML validation passed!! That is valid YAML, but may not meet Kometa's requirements
#Script Not Completing?
charred impBOT
Welcome @hallow ice!
Thanks for being a Kometa Sponsor, we greatly appreciate it! Your ticket will now be diverted to <@&1097919568334311495> and <@&938443185347244033>.
Including meta.log from the beginning is a huge help. Type !logs for more information.
After attaching your log, do not forget to hit the green check boxes when prompted by our bot.
You can press the "Close Post" button above or type /close at any time to close this post.
charred impBOT
You can press the "Close Post" button above or type `/close` at any time to clos...
I cant tell but my last 10 logs all stop at a random point, I attached the one from today but it seems to have been going on for a few weeks.
charred impBOT
hallow ice
I cant tell but my last 10 logs all stop at a random point, I attached the one from today but it seems to have been going on for a few weeks. I can attach more logs, but they all stop at random points. I havent made any config edits in awhile and it was working prior just fine.
Also this is docker and I tried restarting/repulling the containers
devout sparrow
i would check your plex logs and see what might be going on. the logs suddenly end, so this means that the run is either still happening or something outside of Kometa is killing the run
fading arrow
If the log is ending abruptly with no error that generally means that something external to the script is killing it. Out of memory, some OS-level thing killing long-running scripts, that sort of thing.
fading arrow
That log shows it ending abruptly while looking at doing metadata updates after running for only about a minute. There's no error Kometa is seeing and reporting, so somethign outside is killing it, I think. I don't know what that would be offhand, since there seems to be plenty of memory.
hallow ice
That log shows it ending abruptly while looking at doing metadata updates after ...
So I attached all the logs that where saved as well as the docker log and I found one PMS log from 10/13 3AM and the PMM log from the same, I labled them both. I dont see anything stopping the script from running, maybe I missed it, but youll see from the PMM log the cutoff point is random points, i thought the same with the metadata but it doesnt allign with the other cutoff points. Oddly enough, I cant seem to find the corresponding PMS logs from the same date anywhere, but 3PM since the script runs 2x a day. I would think if PMS backend scans/cleanups would cause it then the 3PM would run but it cuts off as well. And obviously since I reached out for some assistance, the PMM gods heard and ran the script completly last night without stopping, even though I didnt change a thing. Ill see if it runs again this afternoon but thought I would post everything anyway
fading arrow
So I attached all the logs that where saved as well as the docker log and I foun...
The problem is that this is entirely outside Kometa.
It's most likely not anything in Plex that's killing it, since that also would have produced something in the Kometa log, some communication error or the like.
Kometa's run is just ending without it having experienced any sort of orderly error, and every time I can recall having seen this it's been something like an OOM condition or something in the OS.
We have seen this happen on appboxes where they're configured to kill scripts after some period of time, but that doesn't seem to fit the facts here.
All of the complete runs include a half-dozen instances where Plex says it can't find something it told Kometa to look for, which ultimately fail with:
[INFO] | 1 | RetryError[<Future at 0x7a8f45eca850 state=finished raised Failed>] |
Not sure what that indicates, but in each case Kometa tries to get that metadata for an item with an ID that Plex would have given it and Plex responds with a 404.
| modules.util.Failed: Item Failed to Load: (404) not_found; (redacted)/library/metadata/1213784 <html><head><title>Not Found</title></head><body><h1>404 Not Found</h1></body></html> |
1213784 there would have come straight from Plex.
hallow ice
The problem is that this is entirely outside Kometa.
It's most likely not anyt...
I see the RetryError all the time since the Kometa switch, it used to be labeled something else with PMM but it was always with a collection list not finding a matching ID or something, but maybe im mistaken.
Ill pull some dmesg logs and see if theres anything happening. The server is on 24/7 and Im still on Ubuntu 22.04 lts and havent upgraded. There are cron scripts running in the background for backups and cleanups but nothing new
fading arrow
There have been some post-rebrand changes to a different means of doing retries, whcih is probably behind the change in the error.
hallow ice
There have been some post-rebrand changes to a different means of doing retries,...
So I pulled some journalctl logs for some times its stopped and found this. The time stamp matches identically when the kometa log stops. Any thougths?
Oct 15 13:01:04 jarvis kernel: traps: python3[551293] general protection fault ip:73b5b147ad18 sp:7ffde1b33130 error:0 in libpython3.11.so.1.0[73b5b1453000+223000]
Oct 15 13:01:04 jarvis audit[551293]: ANOM_ABEND auid=4294967295 uid=0 gid=0 ses=4294967295 subj=docker-default pid=551293 comm="python3" exe="/usr/local/bin/python3.11" sig=11 res=1
fading arrow
It's apparently something below Kometa; seems unlikely to be caused by something within Kometa as the docker image is running all over the place without this problem.
hallow ice
It's apparently something below Kometa; seems unlikely to be caused by something...
Should I move off nightly? Do you think the jefe would know any more?
fading arrow
Also seems unlikely to be nightly, though if you don't have a specific reason to use nightly it wouldn't hurt to move off it.
At this level, all the docker images are identical. They have the same base with the same python infrastructure.
The docker image has been based on the same python:3.11-slim-buster image since December 2022
hallow ice
The docker image has been based on the same `python:3.11-slim-buster` image sinc...
Have you heard of anything between anyone switching from Ubuntu 22.04 or upgrading to 24.04. I’m still on 22.04lts and was waiting to upgrade until more bugs were worked out. Also though I have other docker containers that run automated python scripts with much issue. I’ll check and see if the image cleanup is running fully.
fading arrow
No, the machines I'm running on regularly are 22.04
agile helmBOT
antwanchild used !aenh
@hallow ice, anything else needed here? If not, please type /close and hit enter. Please respond within 24 hours of this message or it will be archived.