I recently fine tuned all the transcoding options to the way i want it but... The immich will use the gpu for like 10ish videos then all of a sudden stop using the gpu and use the cpu to transcode. Would there be any reason why that would occur ? Ive tested the gpu with other tasks like handbrake and plex etc it doesnt miss a beat, only seems to behave weird via immich.
#Transcoding will transcode a handfull of videos on the gpu then wont do anymore and uses cpu
wanton spoke
reef steppeBOT
Checklist
I have...
- :ballot_box_with_check: verified I'm on the latest release(note that mobile app releases may take some time).
- :ballot_box_with_check: read applicable release notes.
- :ballot_box_with_check: reviewed the FAQs for known issues.
- :ballot_box_with_check: reviewed Github for known issues.
- :ballot_box_with_check: tried accessing Immich via local ip (without a custom reverse proxy).
- :ballot_box_with_check: uploaded the relevant information (see below).
- :ballot_box_with_check: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable
(an item can be marked as "complete" by reacting with the appropriate number)
Information
In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:
- Your docker-compose.yml and .env files.
- Logs from all the containers and their status (see above).
- All the troubleshooting steps you've tried so far.
- Any recent changes you've made to Immich or your system.
- Details about your system (both software/OS and hardware).
- Details about your storage (filesystems, type of disks, output of commands like
fdisk -landdf -h). - The version of the Immich server, mobile app, and other relevant pieces.
- Any other information that you think might be relevant.
Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)
If this ticket can be closed you can use the /close command, and re-open it later if needed.
reef steppeBOT
## Checklist
I have...
1. :ballot_box_with_check: verified I'm on the latest rel...
Successfully submitted, a tag has been added to inform contributors. :white_check_mark:
wanton spoke
COMPOSE
ENV
You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables
The location where your uploaded files are stored
LIBRARY_LOCATION=/mnt/DockerHDD/Instances/Storage/Immich/Library/
BACKUPS_LOCATIONS=/mnt/DockerHDD/Instances/Backups/Immich/Backup/
UPLOAD_LOCATION=/mnt/Docker/Instances/Storage/Immich/Upload/
THUMBS_LOCATION=/mnt/Docker/Instances/Apps/Immich/Thumbs/
PROFILE_LOCATION=/mnt/Docker/Instances/Apps/Immich/Profiles/
VIDEO_LOCATION=/mnt/Docker/Instances/Apps/Immich/Encoded-Video/
The location where your database files are stored
DB_DATA_LOCATION=/mnt/Docker/Instances/Apps/Immich/pgdata/
To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
TZ=Australia/Sydney
The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release
Connection secret for postgres. You should change it to a random password
Please use only the characters A-Za-z0-9, without special characters or spaces
DB_PASSWORD=postgres
The values below this line do not need to be changed
###################################################################################
DB_USERNAME=postgres
DB_DATABASE_NAME=immich
and the other yml files are untouched
also on another note, the gpu doesnt hesitate to do machine learning stuff
it stays doing that for as long as it needs to finich the task
lean sphinx
lucid estuary
Please post the server logs
wanton spoke
1 moment
the card is a a770 16gb of vram i dont think it should be running out, im just re running some videos through it again to get a more recent event
ofcourse it seems to be not replicating the issue, let me let it run for a bit longer and ill post here again if i can get it to happen again
im just posting this here to show its currently using the gpu correctly so when it happens i can show the gpu being inactive and the cpu being pined
how do i get the logs you require it just did it
lucid estuary
docker logs immich_server
wanton spoke
uhh
thats all its showing
lemme check something i think i have immich to only spit errors
yer its set to fatal only, sorry one sec
is there a command i can execute to stop all current transcode
so i can get this to happen again with debug for logs
lucid estuary
If you restart the immich server, it’ll start the current transcoding job again from scratch
lucid estuary
It’s fine if it doesn’t use the GPU. I just need logs
I’m not seeing any transcoding jobs that go to CPU. All the jobs in the logs are E2E accelerated
lucid estuary
It’s probably related to #10560, which is probably a driver bug
reef steppeBOT
It’s probably related to #10560, which is probably a driver bug
[Issue] Transcoding stuck (immich-app/immich#10560)
wanton spoke
[Nest] 7 - 11/18/2024, 12:24:02 PM ERROR [Microservices:MediaService] Error occurred during transcoding: ffmpeg was killed with signal SIGABRT
[Nest] 7 - 11/18/2024, 12:24:02 PM ERROR [Microservices:MediaService] Retrying with QSV acceleration disabled
[Nest] 7 - 11/18/2024, 12:24:03 PM DEBUG [Microservices:MediaRepository] ffmpeg -n 10 /usr/bin/ffmpeg -i upload/library/Jayden Olivieri/2021/November/IMG_9673.mov -y -c:v hevc -c:a aac -movflags faststart -fps_mode passthrough -map 0:0 -map 0:1 -refs 6 -tag:v hvc1 -v verbose -vf scale=-2:720,tonemapx=tonemap=hable:desat=0:p=bt709:t=bt709:m=bt709:r=pc:peak=100:format=yuv420p -preset veryslow -b:v 1380k -minrate 690k -maxrate 2000k -pass 1 -passlogfile upload/encoded-video/d58953e2-6cb8-45b6-9e60-a7bea1259423/85/10/851004c4-2c96-4c7a-9a6f-75faebb49734.mp4 -f null /dev/null
does that mean anything ?
lucid estuary
Is this from when you restarted immich?
wanton spoke
oh
potentially
im just reading through the logs
it wouldnt be a ram issue would it ?
i have the immich server limited to 8gb
here i got it to happen again
it transcoded a handfull of videos on the gpu and now is back on the cpu again
lucid estuary
It’s doing thumbnail generation, which is on CPU
wanton spoke
ik
its the quickest way to make this issue occur
i run both at the same time and the gpu stops being used sooner
the gpu eventually stops being used if i run it alone but its after a few more videos
lucid estuary
Huh… that shouldn’t really affect transcoding. I wonder if it depends on certain timing and it’s more likely to encounter the issue if CPU utilization is high.
wanton spoke
yer thats what it seems like
but doesnt really explain when the transcoding is the only thing happening that over time the issue still occurs
lucid estuary
Can you try setting transcoding concurrency to 1?
wanton spoke
thats what it was on last night when i noticed the issue originally
but sure
[Nest] 7 - 11/18/2024, 12:51:45 PM LOG [Microservices:MediaService] Encoding video d5fcfc9b-e7d1-4228-a4ed-8428f7931008 with QSV acceleration
[Nest] 7 - 11/18/2024, 12:51:45 PM DEBUG [Microservices:MediaRepository] ffmpeg -n 10 /usr/bin/ffmpeg -hwaccel qsv -hwaccel_output_format qsv -async_depth 4 -noautorotate -threads 1 -qsv_device /dev/dri/card0 -i upload/library/Destro Media/2023/March/videoplayback (1).webm -y -c:v hevc_qsv -c:a copy -movflags faststart -fps_mode passthrough -map 0:0 -bf 7 -refs 6 -g 256 -tag:v hvc1 -v verbose -vf scale_qsv=-1:720:async_depth=4:mode=hq:format=nv12 -preset 1 -global_quality:v 30 -maxrate 2000k -bufsize 4000k upload/encoded-video/99ac6830-5875-4093-953c-78ee18870368/d5/fc/d5fcfc9b-e7d1-4228-a4ed-8428f7931008.mp4
[Nest] 16 - 11/18/2024, 12:51:55 PM DEBUG [Api:LoggingInterceptor~3zeu5w5r] GET /api/server/ping 200 0.12ms ::ffff:127.0.0.1
[Nest] 7 - 11/18/2024, 12:52:02 PM DEBUG [Microservices:MediaRepository] Transcoding 5.04% done, estimated 5m, 20s remaining for output d5fcfc9b-e7d1-4228-a4ed-8428f7931008.mp4
[Nest] 7 - 11/18/2024, 12:52:19 PM DEBUG [Microservices:MediaRepository] Transcoding 10.10% done, estimated 5m, 2s remaining for output d5fcfc9b-e7d1-4228-a4ed-8428f7931008.mp4
see working fine
only after the existing jobs are zeroed out
lucid estuary
sudo dmesg?
wanton spoke
How much of that output do you want
lucid estuary
sudo dmesg | grep i915
wanton spoke
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
a5716aea9b76 immich_server 80.58% 671.7MiB / 16GiB 4.10% 1.05GB / 35.3MB 8.03GB / 1.34GB 81
also seems to not be related to ram or anything like that
barely using one core
lucid estuary
Yeah this is probably ffmpeg getting stuck
wanton spoke
;-;
lucid estuary
You can look at related issues for media-driver and possibly compute-runtime. Based on the comments in this issue another immich user posted https://github.com/intel/media-driver/issues/1821, it might be a driver bug
/sys/class/drm/card0/error could also have something useful in it
wanton spoke
No error state collected
wanton spoke
how to i check that again
lucid estuary
uname -a
wanton spoke
Linux Destro-Linux-Server 6.11.0-9-generic #9-Ubuntu SMP PREEMPT_DYNAMIC Mon Oct 14 13:19:59 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
6.11
lucid estuary
Oh wow, that’s pretty bleeding edge
wanton spoke
Lol
mainly cause im running z790
thing is though
on 1.19 i have never had this problem
immich 1.19 that is
sorry
1.119
idk if something aside from immich updated at a similar time and has decided to cause this kind of issue
this time no thumbnails
just curious why is it using VAAPI
one other question why does it keep pinging this address ?
DEBUG [Api:LoggingInterceptor~njzi9q9g] GET /api/jobs 200 2.22ms ::ffff:192.168.0.68
::ffff:192.168.0.68 theres no device on that ip
lucid estuary
on 1.19 i have never had this problem
This is good to know actually
wanton spoke
cause i originally migrated from macos to linux on 1.119, i re encoded all videos and thumbnails and everything got redone using the old settings i used to use, i went over all settings last night now that everything is setup properly etc, and noticed that issue
the settings i changed to werent outlandish just higher bitrate really
lucid estuary
So the issue happened with the same settings as on 1.119? Or did you change them before transcoding?
wanton spoke
no the issue never occured on 1.119
if it means anything i changed them back last night and the issue still occured
OLD SETTINGS
"ffmpeg": {
"crf": 21,
"threads": 0,
"preset": "ultrafast",
"targetVideoCodec": "hevc",
"acceptedVideoCodecs": [
"hevc"
],
"targetAudioCodec": "aac",
"acceptedAudioCodecs": [
"aac"
],
"acceptedContainers": [
"mov",
"ogg",
"webm"
],
"targetResolution": "720",
"maxBitrate": "2800k",
"bframes": 10,
"refs": 6,
"gopSize": 0,
"temporalAQ": false,
"cqMode": "icq",
"twoPass": true,
"preferredHwDevice": "card2",
"transcode": "all",
"accel": "qsv",
"accelDecode": true,
"tonemap": "hable"
NEW SETTINGS
"ffmpeg": {
"accel": "qsv",
"accelDecode": true,
"acceptedAudioCodecs": [
"aac"
],
"acceptedContainers": [
"mov",
"ogg",
"webm"
],
"acceptedVideoCodecs": [
"hevc"
],
"bframes": -1,
"cqMode": "icq",
"crf": 30,
"gopSize": 0,
"maxBitrate": "2000k",
"preferredHwDevice": "card0",
"preset": "veryslow",
"refs": 6,
"targetAudioCodec": "aac",
"targetResolution": "720",
"targetVideoCodec": "hevc",
"temporalAQ": false,
"threads": 0,
"tonemap": "hable",
"transcode": "all",
"twoPass": true
thats from the exported json
wanton spoke
it does seem that if the cpu is given a task such as thumbnails it greatly reduces how many videos the transcodes get through before it goes to cpu encoding
the encoder utilization also greatly drops
goes from like 60% down to 10-30
without thumbs
with thumbs going at the same time as one video encode
lucid estuary
I think the only maybe relevant change in 1.120.0 was that tonemap_mode=lum was added to the command. Try disabling tone-mapping?
wanton spoke
i allocated 4 cores to the server and limited all jobs concurrency to 1 so far its been running fine
the encoder is staying around 60ish percent usage
its done close to 100 videos now
im going to delete the encoded-video folders just to prove it definitely is working
and restart the task of doing all videos
starting at 12790 waiting
ill check back in like 30 minutes to be sure its all good
atleast for the most part everything seems fine aslong as the cpu cores alocated to immich server arent overloaded for some reason that causes the gpu to stop being utilized for video transcoding. but right now im doing both thumbs and video and its fine
its literally done over 400 videos and still fine
lucid estuary
Okay, so it’s definitely something to do with tone-mapping
lucid estuary
Oh, you just limited cpu?
wanton spoke
well increased allocated cpu cores but limited the tasks all to 1
i believe if i set it to do something outlandish like 100 thumbnails at once it would cause the gpu to stop being used
as long as cpu util doesnt max out on the container the gpu continues to work as normal
once the container maxes out the allocated cpu cores it stops using the gpu for transcodes and the cpu stays maxed out cause of transcoding on the cpu
its literally done like 800 videos now without any issues
lucid estuary
My hypothesis is that as it’s processing the video, it’s queueing events and data with the expectation that they’ll be processed within a certain timeframe. If the CPU is busy and can’t sift through those events quickly enough, the GPU gets into a bad state. And since the ffmpeg process is still alive (there’s no explicit error), none of this can be freed by the kernel
wanton spoke
solution ?
lucid estuary
idk lol
wanton spoke
can it be possible in a future release you can limit threads for thumbnail generation
lucid estuary
It’s an upstream problem
wanton spoke
dunno if someone else can recreate the issue or if maybe its me specific
lucid estuary
You can look at related issues for media-driver and possibly compute-runtime. Ba...
You can share the behavior you’re seeing, the system specs and the FFmpeg command in this issue. Or you can make a different issue. I think the fix for this will have to come from there
And of course share any other info they’d like
wanton spoke
i still think its an immich issue though cause any other tasks that utilize the gpu i tried to overload it or make it miss behave but only immich can cause weird behaviour
cause the second i set thumbnail concurrency to something that pins the cores to max the gpu will stop being used for video encoding
lucid estuary
It might be specific to the particular environment in the container, like the driver version, etc. But it’s ultimately not our bug to fix
lucid estuary
Yeah, that should be fine. The niceness setting might also be relevant here. We set FFmpeg’s niceness to 10 so it only uses “spare” CPU cycles. That might be exacerbating this
wanton spoke
it got to 1000 ish videos before failing again
ive disabled tone mapping like you asked and it already has done 1500 videos
and still working
wanton spoke
ok nope still happened
got to around 3500 videos done and it still fell back to using the cpu
wanton spoke
im kinda losing my mind
ive troubleshooted so many scenarios
videos work example
[Nest] 7 - 11/19/2024, 2:35:29 AM LOG [Microservices:MediaService] Encoding video d0b7dd95-41e8-420f-aa4e-545ee534ab9d with QSV acceleration
[Nest] 7 - 11/19/2024, 2:35:29 AM DEBUG [Microservices:MediaRepository] ffmpeg -n 10 /usr/bin/ffmpeg -hwaccel qsv -hwaccel_output_format qsv -async_depth 4 -noautorotate -threads 1 -qsv_device /dev/dri/card0 -i upload/library/Jayden Olivieri/2019/March/IMG_9663.mov -y -c:v hevc_qsv -c:a aac -movflags faststart -fps_mode passthrough -map 0:0 -map 0:1 -bf 7 -refs 5 -g 256 -tag:v hvc1 -v verbose -vf scale_qsv=-1:1080:async_depth=4:mode=hq:format=nv12 -preset 1 -global_quality:v 25 -maxrate 3000k -bufsize 6000k upload/encoded-video/d58953e2-6cb8-45b6-9e60-a7bea1259423/d0/b7/d0b7dd95-41e8-420f-aa4e-545ee534ab9d.mp4
[Nest] 7 - 11/19/2024, 2:35:29 AM DEBUG [Microservices:MediaRepository] Transcoding 31.79% done, estimated 10s remaining for output 03731f72-cb2c-467c-86c6-7c722e89072d.mp4
[Nest] 7 - 11/19/2024, 2:35:29 AM DEBUG [Microservices:MediaRepository] Transcoding 55.94% done for output de8b6bf4-7e2c-482b-830e-9c145442bc27.mp4
[Nest] 7 - 11/19/2024, 2:35:30 AM DEBUG [Microservices:MediaRepository] Transcoding 77.36% done for output d0b7dd95-41e8-420f-aa4e-545ee534ab9d.mp4
[Nest] 7 - 11/19/2024, 2:35:30 AM DEBUG [Microservices:MediaRepository] Transcoding 100.00% done for output d0b7dd95-41e8-420f-aa4e-545ee534ab9d.mp4
[Nest] 7 - 11/19/2024, 2:35:30 AM LOG [Microservices:MediaService] Successfully encoded d0b7dd95-41e8-420f-aa4e-545ee534ab9d
and then
wanton spoke
still same issues ive tried everything i can possibly think of
only solution i can think of is its something immich related i have 0 control over
machine learning tasks run perfect, one video here or there perfect, once i leave it to reencode the entire library it gets to a certain point and stops using gpu and moves all work to the cpu
idk what else to say
lucid estuary
Report about it upstream if you want it to be fixed. Not much I can do
wanton spoke
? What do you mean upstream
lucid estuary
This commit might be related. It's in 1.120.2 (which uses jellyfin-ffmpeg 7.0.2-6) but not 1.120.1 (uses 7.0.2-5). Can you try 1.120.1?
reef steppeBOT
[This](https://github.com/jellyfin/jellyfin-ffmpeg/pull/495/files) commit might ...
[Pull Request] Use dynamic pool for VPL QSV hwupload (jellyfin/jellyfin-ffmpeg#495)
wanton spoke
sorry i went to bed
can i safely downgrade to 1.20.1 without breaking anything ?
- IMMICH_VERSION=1.120.1
?
sorted
trying nowe
if literally just downgrading the server fixed it imma just casually die
lucid estuary
can i safely downgrade to 1.20.1 without breaking anything ?
Yeah should be totally fine
wanton spoke
i deleted all folders in encoded and thumbs so i know 100% its working
so far seems to actually be working fine
its done about 1000 videos so far
Nope it still happened
I did however realise something
its the same video it failed on
its actually an insignificant video so i might just delete it
lucid estuary
Oh then it’s kinda working fine then? Just a few videos that it’s failing on
wanton spoke
but deleted them from the library atm
no
i could be wrong
but its a bit coincidental it fails on the same videos every time
when i press all is the cycle the same ?
press transcode all i mean
does it start from the same begining every time ?
lucid estuary
It’s ordered by when the assets were imported into immich
wanton spoke
so when i press it the cycle is the same everytime ?
lucid estuary
Yes
wanton spoke
also manually refreshing those videos works fine
hmm
FFmpeg was terminated with signal SIGABRT, indicating an abnormal termination due to a runtime error. This signal is typically sent by the operating system when a process attempts to access memory that is not valid or has been freed.
lucid estuary
Is this from the manual refresh or before that?
wanton spoke
that error i sent was during "all"
lucid estuary
At this point you have enough useful logs and evidence of weird behavior that the most effective option is to open an issue to jellyfin-ffmpeg. Provide the logs and FFmpeg command, describe the behaviors you’re seeing and maybe share one of the problematic videos. They’ll let you know if this is their bug or guide you if this is further upstream
wanton spoke
its not one of the videos tho
i just checked that
those videos ended up being hardware transcoded by the gpu perfectly fine
lucid estuary
Okay, then you can explain that and go from there
wanton spoke
thats why i asked about the cyle if its the same every time, with tone mapping disabled and the transcoder set to 1 video at a time it gets through like 4000 videos before erroring out
lucid estuary
It’s hard to say what the root cause is, but it seems like an internal edge case that needs to be looked at by someone more intimate with FFmpeg internals
wanton spoke
so i spoke to some people over at ffmpeg github
they said my gpu driver is crashing due to heavy load, but its set to 1 video for transcoding and this issue only happens with immich as i tested with handbrake numerous times and i cant get the same issue to occur
wanton spoke
if i run machine learning via immich it does everything at full load no issues at all, ive literally been working on this for over 4 days now
and i have narrowed down to a video that has the issue every single time
i even converted the video with handbrake re uploaded it to immich same thing happens hardware encoding isnt used and uses the cpu
i cannot however replicate this issue with any other piece of software other then immich
there has to be something immich is doing causing this problem i cannot think of any other problem
lucid estuary
If they think it’s a GPU driver issue, then report about it in the media-driver repo. It’d be interesting to see if installing a different version of media-driver-va-non-free in the container might fix the issue
wanton spoke
I updated that, even tried removing and reinstalling, the free and non free version tried both
wanton spoke
Because I’m on the verge of pulling my hair out
To rule out anything to do with my install, I’m doing a temporary fresh install on a spare drive I have
I’ll let you know if the issue persists
wanton spoke
Yep got the identical error
Just did a fresh install of everything
Must be a driver issue or again something immich is doing I have no other things to try or do
From what I can find too
Is intel updated the driver around the 30th of October
Which I guess lines up
And for what ever reason my driver is a few builds newer then the latest version n the changelog
Very weird
attempted to downgrade video driver downgraded successfully everything else still works but crashes when immich trys to encode video
just tried on igpu and dgpu
interesting the file that it keeps crashing on if i disable tone mapping it works fine
that is a headache and a half, but i still know eventually this issue still occurs with tone mapping disabled
lucid estuary
Tone-mapping does increase the GPU load
wanton spoke
yer
the issue will still happen on certain videos im running all again to find one it happens with
the file it was crashing with tho is litterally a 10 second clip i recorded
but it encodes an hr long video fine both on the same phone same video settings
doesnt make sense to me
lucid estuary
To be clear, I’m happy to make a change that would fix this for you if it doesn’t have a negative effect on others. But this is not our bug and we don’t have the bandwidth to do a deep investigation of it. Make an issue about it in media-driver if you want it to be fixed
wanton spoke
thats fine, i just dont understand what changed from 1.119 to now to cause this
cause i have ran all on the library before with not too different settings and it ran on the gpu fine with tone mapping and all
it just doesnt make sense thats all
the change you make couldnt it be like a toggle for people that need it ? what is the change anyway
im just curious
if it means anything i just tried hdr tone mapping via plex, no issues
dunno if thats the same situation but eh
lucid estuary
The changes include setting peak=100 during tone-mapping (shouldn’t have any relevance to this), setting tonemap_mode=lum (very unlikely to be relevant), a small dependency bump from FFmpeg 7.0.2-5 to 7.0.2-6, and I guess the driver was updated?
wanton spoke
if you think it will fix it iguess
the exact same video file i just tried on plex too and it transcodes and tone maps perfectly fine
literally only happens via immich
lucid estuary
This is all very sensitive to dependency versions and the specifics of the FFmpeg command
wanton spoke
yer i understand
im just letting you know everything im testing
if you want do you want me to privately send you the video file ?
and you can test it yourself ?
lucid estuary
Sure I can test it, but it would be on an iGPU and not on an A770
wanton spoke
ive tested it on an igpu aswell
same error occurs
cause i have
A770
and
igpu of the 12900ks
can i strip location data some how from the file before i send it
dw found out how
i wonder imma try ffmpeg directly on ubuntu with an identical copy of the command it uses in immich and see what happens
the small things you plan to change, you cant make them options in immich itself so people dont necesarily have to all use them ?
lucid estuary
What do you mean by things I plan to change?
lucid estuary
The changes include setting `peak=100` during tone-mapping (shouldn’t have any r...
If you mean this, these are changes between 1.119 and 1.120
wanton spoke
oh
righttt you were replying to that
i thought you were saying changes you plan to make
sorry
lucid estuary
Oh, before that I meant that if I knew which change would fix your issue, I would make the change if it didn’t have side effects for others
lucid estuary
It goes from being an issue affecting your particular environment to certain videos just not working
wanton spoke
lol
pray
ill dm you
the file might be appe pro res does that make a difference ?
lucid estuary
I don’t think prores can be hardware decoded
wanton spoke
could that be the issue then ?
still doesnt explain that i used to run all before and it never fell back to the cpu for encoding but eh idk at this point
lucid estuary
*unless using videotoolbox
lucid estuary
could that be the issue then ?
Definitely
wanton spoke
did you download the file ?
see if it happens for you
ill disable hardware decoding and try transcode again with tone mapping
theres no way
its the hardware decoding causing the issue it looks like
FML
this entire time its just hardware decoding
bruh
i just re enabled it and it crashed again
FML
this entire time
lol
i enabled tone mapping and everything it works flawlessly
farrrrr out
lucid estuary
lmao
wanton spoke
does that mean anything for you tho
lucid estuary
There is a PR that would make hw decode + encode fall back to hw encode first before going full software
wanton spoke
yer well what ever error occurs makes it decode and encode via software when hw decode is enabled for this specific video
wait
lucid estuary
Yeah so if hw decode is disabled, it will work the first time around without falling back
But if it's enabled, it becomes all or nothing
wanton spoke
but also doesnt make sense that hw decode was enabled with tone map disabled and it decodes/encodes fine
so
hw decode enabled, tone map enabled fails
hw decode enabled, tone map disabled works
hw decode disabled, tone map enabled works
hw decode disabled, tone map disabled works
lucid estuary
tbh I’m not sure why it would work at all with hw decoding
But it’s clearly not something the filters can handle properly regardless
Is this the one where it errors with std::out_of_range?
wanton spoke
I can’t check anymore I’m not at home I went out
Ill grab the error I posted to GitHub 1 sec
This is the error I sent
It includes a successful pass
This is with both hw decode and tone mapping enabled
Can’t send on my phone here’s the link
I have the server currently doing all videos again to see if the error still occurs with a different video file potentially
So far so good
It’s already done 1500 videos with no signs of issues yet
Also is a worse case scenario too
It’s doing 2 tasks for videos and I’m doing all thumbs at once too (10 at a time))
lucid estuary
Nice
If you run the FFmpeg command manually for that prores file and add :format=p010 to the scale_qsv options, does it work?
wanton spoke
7500 now no errors lol
Willlllllllldddd
wanton spoke
11000 ish
I have noticed the encoder utilisation dropped by quite a bit a while ago. I mean it’s still being used and has been
Encoder is maxing out at 20% util for some videos
lucid estuary
Decoding is probably bottlenecking it
wanton spoke
Yer seems to pin cpu at 100% for decoding
It’s up to the last 310 videos and everything worked perfectly so far
wanton spoke
Aside from being very notably slower and barely utilising the entire video encoder now, it did all videos with absolutely 0 errors or issues. I still wonder why all of a sudden using the hw decode causes issues in immich but not in something like handbrake
Oh well at least it’s working now
lucid estuary
If you run the FFmpeg command manually for that prores file and add `:format=p01...
Can you try this? It might let hw decoding work on those prores videos
wanton spoke
Can you try this? It might let hw decoding work on those prores videos
can you give me an example of the whole command please you want me to run
can i just adjust this ?
ffmpeg -n 10 /usr/bin/ffmpeg -hwaccel qsv -hwaccel_output_format qsv -async_depth 4 -noautorotate -threads 1 -qsv_device /dev/dri/card0 -i upload/library/redacted/2021/November/IMG_9575.mov -y -c:v hevc_qsv -c:a aac -movflags faststart -fps_mode passthrough -map 0:0 -map 0:1 -bf 7 -refs 5 -g 256 -tag:v hvc1 -v verbose -vf scale_qsv=1080👎async_depth=4:mode=hq,hwmap=derive_device=opencl,tonemap_opencl=desat=0:format=nv12:matrix=bt709:primaries=bt709:transfer=bt709:range=pc:tonemap=hable:tonemap_mode=lum:peak=100,hwmap=derive_device=qsv:reverse=1,format=qsv -preset 1 -global_quality:v 25 -maxrate 3000k -bufsize 6000k upload/encoded-video/d58953e2-6cb8-45b6-9e60-a7bea1259423/b4/10/b410223b-ddad-43f8-81e5-47e82adb1c84.mp4
lucid estuary
You see how you have :mode=hq there? Just add :format=p010 after it
lucid estuary
Yes, that comes later
wanton spoke
alright
like this ?
async_depth=4:mode=hq,:format=p010,hwmap=derive_device=opencl
or no ,
lucid estuary
No comma
wanton spoke
okeh
[AVFormatContext @ 0x619503502440] Unable to choose an output format for '10'; use a standard extension for the filename or specify the format manually.
[out#0 @ 0x619503562c00] Error initializing the muxer for 10: Invalid argument
Error opening output file 10.
Error opening output files: Invalid argument
[AVIOContext @ 0x619503564700] Statistics: 501128 bytes read, 2 seeks
wanton spoke
discord changing stuff
1 sec
ffmpeg -n 10 /usr/bin/ffmpeg -hwaccel qsv -hwaccel_output_format qsv -async_depth 4 -noautorotate -threads 1 -qsv_device /dev/dri/card2 -i /home/destro/Desktop/IMG_9673.mov -y -c:v hevc_qsv -c:a aac -movflags faststart -fps_mode passthrough -map 0:0 -map 0:1 -bf 7 -refs 2 -g 256 -tag:v hvc1 -v verbose -vf hwmap=derive_device=opencl,tonemap_opencl=desat=0:format=nv12:matrix=bt709:primaries=bt709:transfer=bt709:range=pc:tonemap=hable:tonemap_mode=lum:peak=100,hwmap=derive_device=qsv:reverse=1,format=qsv -preset 1 -global_quality:v 24 -maxrate 3000k -bufsize 6000k /home/destro/Desktop/exportedversion.mp4
i just got the latest command its using currently from immich
i think -n 10 is the issue ?
i removed the begining part
still errors
this command works
ffmpeg -hwaccel qsv -hwaccel_output_format qsv -i /home/destro/Desktop/IMG_9673.mov -vf "vpp_qsv=tonemap=1" -c:v hevc_qsv -crf 23 -c:a aac /home/destro/Desktop/exportedversion.mp4
ok ok ok
adjusted it a little to be on the arc gpu and this works
ffmpeg -hwaccel qsv -hwaccel_output_format qsv -qsv_device /dev/dri/card2 -i /home/destro/Desktop/IMG_9673.mov -vf "vpp_qsv=tonemap=1" -c:v hevc_qsv -crf 23 -c:a aac -maxrate 3000k -bufsize 6000k /home/destro/Desktop/exportedversion.mp4
idk what to add to the command lol
made a heap of changes to the command still works flawlessly
wanton spoke
ffmpeg -hwaccel qsv -hwaccel_output_format qsv -threads 4 -qsv_device /dev/dri/card2 -i /home/destro/Desktop/IMG_9673.mov -map 0:0 -map 0:1 -bf 7 -refs 2 -g 256 -tag:v hvc1 -vf "scale_qsv=1920:1080:mode=hq,vpp_qsv=tonemap=1,format=nv12" -c:v hevc_qsv -crf 23 -c:a aac -movflags faststart -fps_mode passthrough -preset 1 -global_quality:v 24 -maxrate 3000k -bufsize 6000k /home/destro/Desktop/exportedversion.mp4
this works still
wanton spoke
okokokokok i made all the adjustments to basically make the same file immich does except with hw encode and decode via ffmpeg with 0 issues
via qsv
ffmpeg -hwaccel qsv -hwaccel_output_format qsv -async_depth 4 -noautorotate -threads 1 -qsv_device /dev/dri/card2 -i /home/destro/Desktop/IMG_9673.mov -y -c:v hevc_qsv -c:a aac -movflags faststart -fps_mode passthrough -map 0:0 -map 0:1 -bf 7 -refs 5 -g 256 -tag:v hvc1 -v verbose -vf "scale_qsv=1920:1080:mode=hq,vpp_qsv=tonemap=1,format=nv12" -preset 1 -global_quality:v 23 -maxrate 3000k -bufsize 6000k /home/destro/Desktop/exportedversion.mp4
so it seems for the most part the issue i keep being able to generate is tonemap_opencl
if i use intels hardware option for tonemapping
it works flawlessly
vpp_qsv=tonemap=1
can i make a suggestion, maybe in the hardware acceleration section allow for a manual override ? like keep all the options as is but have an advanced users option that allows us to put our own ffmpeg options ?
that could alleviate future issues like this maybe ?
idek i just want this to work with the hardware i paid good money for
so i created a dumby account, added the original file to that account let immich encode it then replaced the ecoded file with my version and it works and streams flawlessly looks normal and everything
dunno what else to say but immich is borked for qsv encoding with the methods it uses atleast for me it is
wanton spoke
sorry for basically spamming this chat, but now i have tried like 20 different ways of transcoding this same video, to put everything simply immich will only work if hw decode is off because the video in question isnt pro res i have found, it was just 4k 60fps hdr
lucid estuary
It looks like you tried literally everything but setting format=p010…
lucid estuary
Can you share the command?
wanton spoke
ffmpeg -hwaccel qsv -hwaccel_output_format qsv -async_depth 4 -noautorotate -threads 4 -qsv_device /dev/dri/card2 -i /home/destro/Desktop/IMG_9673.mov -y -c:v hevc_qsv -c:a aac -b:a 128k -movflags faststart -fps_mode passthrough -map 0:0 -map 0:1 -bf 7 -refs 5 -g 256 -tag:v hvc1 -v verbose -vf "scale_qsv=1920:1080:mode=hq,vpp_qsv=tonemap=1,format=nv12" -preset 1 -global_quality:v 23 -maxrate 3000k -bufsize 6000k /home/destro/Desktop/exportedversion.mp4
where do you want it added to this
exactly to hq ?
lucid estuary
After hq, before the comma
lucid estuary
Yup
lucid estuary
Nice. Now can you make that same change to the command that immich runs?
wanton spoke
via docker exec ?
lucid estuary
That would be fine yes
wanton spoke
ive been trying to use that cause i saw you comment someone to try it in another thread
it thinks the location doesnt exist
lucid estuary
Only use the /usr/bin/ffmpeg part of the command onward
wanton spoke
docker exec immich_server ffmpeg -hwaccel qsv -hwaccel_output_format qsv -async_depth 4 -noautorotate -threads 4 -qsv_device /dev/dri/card2 -i /upload/library/ad9a1bff-8118-4c5d-a300-614068f94340/2021/November/IMG_9673.mov -y -c:v hevc_qsv -c:a aac -b:a 128k -movflags faststart -fps_mode passthrough -map 0:0 -map 0:1 -bf 7 -refs 5 -g 256 -tag:v hvc1 -v verbose -vf "scale_qsv=1920:1080:mode=hq:format=p010,vpp_qsv=tonemap=1,format=nv12" -preset 1 -global_quality:v 23 -maxrate 3000k -bufsize 6000k /upload/encoded-video/ad9a1bff-8118-4c5d-a300-614068f94340/c5/6f/c56f0c8f-8be5-460a-b89b-507b8bc527f2.mp4
oh
does it need extra stuff
lucid estuary
This is a different command. Immich doesn’t use VPP tone-mapping
It should use tonemap_opencl
wanton spoke
oh well that complicates it slightly
i cant use tonemap opencl
it failed every time i attempted it
idk why
lucid estuary
This is making a change to the command and testing to see if it works with that change
wanton spoke
so ill just replace vpp
lucid estuary
It’ll be better to start with the immich command and just make that change
wanton spoke
yer imma do that
1 moment
hq isnt in the command the latest version immich uses
docker exec immich_server ffmpeg -n 10 /usr/bin/ffmpeg -hwaccel qsv -hwaccel_output_format qsv -async_depth 4 -noautorotate -threads 1 -qsv_device /dev/dri/card2 -i /upload/library/ad9a1bff-8118-4c5d-a300-614068f94340/2021/November/IMG_9673.mov -y -c:v hevc_qsv -c:a aac -movflags faststart -fps_mode passthrough -map 0:0 -map 0:1 -bf 7 -refs 2 -g 256 -tag:v hvc1 -v verbose -vf hwmap=derive_device=opencl,tonemap_opencl=desat=0:format=nv12:matrix=bt709:primaries=bt709:transfer=bt709:range=pc:tonemap=hable:tonemap_mode=lum:peak=100,hwmap=derive_device=qsv:reverse=1,format=qsv -preset 1 -global_quality:v 24 -maxrate 3000k -bufsize 6000k /upload/encoded-video/ad9a1bff-8118-4c5d-a300-614068f94340/c5/6f/c56f0c8f-8be5-460a-b89b-507b8bc527f2.mp4
where should i modify that to add your format option
lucid estuary
Did you change the target resolution or something?
wanton spoke
the command i posted earlier was from a log from a while ago
lucid estuary
Or the video could just be the same resolution as the target
wanton spoke
4k to 1080 ? lol
ohhhhh
wait
it was a 720p video
that i tested today and got the error with
would that be why hq disapeared from the command
lucid estuary
Yes, because that’s an option for the scale_qsv filter that isn’t needed in this case
lucid estuary
docker exec immich_server ffmpeg -n 10 /usr/bin/ffmpeg -hwaccel qsv -hwaccel_out...
For this command, try adding scale_qsv=format=p010, before hwmap
wanton spoke
docker exec immich_server ffmpeg -n 10 /usr/bin/ffmpeg -hwaccel qsv -hwaccel_output_format qsv -async_depth 4 -noautorotate -threads 1 -qsv_device /dev/dri/card2 -i /upload/library/ad9a1bff-8118-4c5d-a300-614068f94340/2021/November/IMG_9673.mov -y -c:v hevc_qsv -c:a aac -movflags faststart -fps_mode passthrough -map 0:0 -map 0:1 -bf 7 -refs 2 -g 256 -tag:v hvc1 -v verbose -vf scale_qsv=format=p010,hwmap=derive_device=opencl,tonemap_opencl=desat=0:format=nv12:matrix=bt709:primaries=bt709:transfer=bt709:range=pc:tonemap=hable:tonemap_mode=lum:peak=100,hwmap=derive_device=qsv:reverse=1,format=qsv -preset 1 -global_quality:v 24 -maxrate 3000k -bufsize 6000k /upload/encoded-video/ad9a1bff-8118-4c5d-a300-614068f94340/c5/6f/c56f0c8f-8be5-460a-b89b-507b8bc527f2.mp4
this ?
lucid estuary
Yup
wanton spoke
[in#0 @ 0x5795b14fb340] Error opening input: No such file or directory
Error opening input file /upload/library/ad9a1bff-8118-4c5d-a300-614068f94340/2021/November/IMG_9673.mov.
Error opening input files: No such file or directory
definitely exists
i did docker exec cd all the way to the file
docker exec immich_server ls upload/library/ad9a1bff-8118-4c5d-a300-614068f94340/2021/November
shows IMG_9673.mov
lucid estuary
If it’s easier, you can run docker exec -it immich_server bash, cd to that path and run the command with relative paths
The command just needs to use the right input file
wanton spoke
ill just get it to place the file in the same directory
ILL DM YOU THE ERROR IT CONTAINS SOMETHING I DONT WANT ON A PUBLIC FORUM
whoops caps
this command worked
ffmpeg -hwaccel qsv -hwaccel_output_format qsv -async_depth 4 -noautorotate -threads 4 -qsv_device /dev/dri/card2 -i IMG_9673.mov -y -c:v hevc_qsv -c:a aac -b:a 128k -movflags faststart -fps_mode passthrough -map 0:0 -map 0:1 -bf 7 -refs 5 -g 256 -tag:v hvc1 -v verbose -vf "scale_qsv=1920:1080:mode=hq:format=p010" -preset 1 -global_quality:v 23 -maxrate 3000k -bufsize 6000k c56f0c8f-8be5-460a-b89b-507b8bc527f2.mp4
i even moved the encoded file made by docker exec to the correct directory and it plays fine on my phone and on a web page
logged out logged in still works
redid the command containing everything from the docker command and it fails
/usr/bin/ffmpeg -hwaccel qsv -hwaccel_output_format qsv -async_depth 4 -noautorotate -threads 1 -qsv_device /dev/dri/card2 -i IMG_9673.mov -y -c:v hevc_qsv -c:a aac -movflags faststart -fps_mode passthrough -map 0:0 -map 0:1 -bf 7 -refs 5 -g 256 -tag:v hvc1 -v verbose -vf scale_qsv=1080:async_depth=4:mode=hq:format=p010,hwmap=derive_device=opencl,tonemap_opencl=desat=0:format=nv12:matrix=bt709:primaries=bt709:transfer=bt709:range=pc:tonemap=hable:tonemap_mode=lum:peak=100,hwmap=derive_device=qsv:reverse=1,format=qsv -preset 1 -global_quality:v 25 -maxrate 3000k -bufsize 6000k c56f0c8f-8be5-460a-b89b-507b8bc527f2.mp4
wanton spoke
this command worked
ffmpeg -hwaccel qsv -hwaccel_output_format qsv -async_depth...
with this the file is like 10x the size though not of the original but of what it normally would encode to
via docker with hw decode disabled file is 4mb
with that command the file is 22mb
/usr/bin/ffmpeg -init_hw_device qsv=hw,child_device=/dev/dri/card2 -filter_hw_device hw -i upload/library/ad9a1bff-8118-4c5d-a300-614068f94340/2021/November/IMG_9673.mov -y -c:v hevc_qsv -c:a aac -movflags faststart -fps_mode passthrough -map 0:0 -map 0:1 -bf 7 -refs 2 -g 256 -tag:v hvc1 -v verbose -vf tonemapx=tonemap=hable:desat=0:p=bt709:t=bt709:m=bt709:r=pc:peak=100:format=yuv420p,hwupload=extra_hw_frames=64,scale_qsv=-1:1080:mode=hq:format=p010 -preset 1 -global_quality:v 24 -maxrate 3000k -bufsize 6000k upload/encoded-video/ad9a1bff-8118-4c5d-a300-614068f94340/c5/6f/c56f0c8f-8be5-460a-b89b-507b8bc527f2.mp4
running that file becomes 26mb
/usr/bin/ffmpeg -init_hw_device qsv=hw,child_device=/dev/dri/card2 -filter_hw_device hw -i upload/library/ad9a1bff-8118-4c5d-a300-614068f94340/2021/November/IMG_9673.mov -y -c:v hevc_qsv -c:a aac -movflags faststart -fps_mode passthrough -map 0:0 -map 0:1 -bf 7 -refs 2 -g 256 -tag:v hvc1 -v verbose -vf tonemapx=tonemap=hable:desat=0:p=bt709:t=bt709:m=bt709:r=pc:peak=100:format=yuv420p,hwupload=extra_hw_frames=64,scale_qsv=-1:1080:mode=hq:format=p010:format=nv12 -preset 1 -global_quality:v 24 -maxrate 3000k -bufsize 6000k upload/encoded-video/ad9a1bff-8118-4c5d-a300-614068f94340/c5/6f/c56f0c8f-8be5-460a-b89b-507b8bc527f2.mp4
added nv12 back
gpu encoder util was 10% at best
wanton spoke
i believe i might have been doing it slightly wrong for what you had wanted
i just got a copy with hw decode enabled, now i copied it and added your format=p010 to the command im about to test it
lucid estuary
Try tonemap_mode=itp instead of tonemap_mode=lum, plus the p010 change
wanton spoke
DIDNT SEE THAT MESSAGE WILL TRY NOW
whoops caps again
i came here just to mention mesa va drivers had an update, didnt fix anything immich wise
every other method still works as expected
the output file is 0 bytes and not viewable
it still just keeps saying std out of range
lucid estuary
Hmm that's a shame. I don't have many other suggestions other than trying different driver versions in the container (install, then restart without recreating the container). I don't want to support VPP
wanton spoke
:/
i still am genuinely confused as to what has changed either in immich or a driver that is now causing this problem
and the issue isnt recreatable outside of immich
lucid estuary
Immich's transcoding stack is closer to Jellyfin's than Plex's. Jellyfin uses tonemap_opencl as well, so if the video plays there it could give a lead as to what they're doing that immich isn't
wanton spoke
what is the big issue with what ever vpp is
want me to make a test install of jellyfin ?
lucid estuary
That would be helpful yes
lucid estuary
VPP doesn't support any of the tone-mapping options we have, is specific to Intel devices and produces a worse output. Even Jellyfin docs don't really recommend it. Device-agnostic filters like those based on OpenCL or Vulkan are much better
wanton spoke
i see
LOLLLLLL, jelly fin works perfectly
i configured the transcoder to use qsv etc hits the gpu as expected and uses gpu for decode and encode etc
xDDDDDD
um
lol
can i dm you something that is literally going to make me pop a blood vessel
lucid estuary
Sure
wanton spoke
Moral of the story for anyone who stubbles across this issue when decoding while utilizing intel qsv......
when setting the hardware device manually in immich
Set it as /dev/dri/renderDxxx
the xx will be specific numbers that correspond to your config
reef steppeBOT
This thread has been closed. To re-open, use the button below.
lucid estuary
This should be fixed with #14455
reef steppeBOT
This should be fixed with #14455
[Pull Request] fix(server): always set transcoding device, prefer renderD* (immich-app/immich#14455)