ERROR [Error: ENOENT: no such file or directory, access 'upload/thumbs/ Unable to send file: Error

[summer rain]

v1.91.4

Hi all 🙂

I was making some disk changes and accidentally moved a bunch of files out of their current directories into new ones, then after a day found the issue and moved them back.

Basically, I have 10 disks in a mergerfs pool.
I took 1 of those disks and copied the files to /photos/photos instead of /photos. This caused various files in /library, /upload, /encoded-video, and /thumbs to be in the wrong location temporarily.

My immich server logs are now full of these errors - and I sometimes have thumbnail issues and broken images now as a result.

[Nest] 7 - 12/27/2023, 8:42:57 AM ERROR [Error: ENOENT: no such file or directory, access 'upload/library/0c5a6997-7433-424f-a12e-189f5316ef04/2023/2023-12-25/IMG_1221.PNG'] Unable to send file: Error [Nest] 7 - 12/27/2023, 8:42:57 AM WARN [ExpressAdapter] Content-Type doesn't match Reply body, you might need a custom ExceptionFilter for non-JSON responses

I regenerated ALL thumbnails which fixed up most blurred images and broken links, but I still have these errors and probably a lot of broken things that I can't see in my library (600GB).

Is there a safe way to just wipe all thumbnails and repair the database or whatever is the issue? Time is not an issue. Happy to regenerate anything required.

[uneven oriole]

:wave: Hey @summer rain,

Thanks for reaching out to us. Please follow the recommended actions below; this will help us be more effective in our support effort and leave more time for building Immich .

References

• Container Logs: docker compose logs docs
• Container Status: docker compose ps docs
• Reverse Proxy: https://immich.app/docs/administration/reverse-proxy

Checklist

1. :ballot_box_with_check: I have verified I'm on the latest release(note that mobile app releases may take some time).
2. :ballot_box_with_check: I have read applicable release notes.
3. :ballot_box_with_check: I have reviewed the FAQs for known issues.
4. :ballot_box_with_check: I have reviewed Github for known issues.
5. :ballot_box_with_check: I have tried accessing Immich via local ip (without a custom reverse proxy).
6. :ballot_box_with_check: I have uploaded the relevant logs, docker compose, and .env files using the buttons below or the /upload command.
7. :ballot_box_with_check: I have tried an incognito window, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable

(an item can be marked as "complete" by reacting with the appropriate number)

If this ticket can be closed you can use the /close command, and re-open it later if needed.

[summer rain]

Are these paths also right? There is no upload/thumbs directory (never was).

`ENOENT: no such file or directory, access 'upload/thumbs/0c5a6997-7433-424f-a12e-189f5316ef04/89/1e/891ed5a1-0f93-4dbc-a3b6-0c208066bb85.webp'

ENOENT: no such file or directory, access 'upload/library/0c5a6997-7433-424f-a12e-189f5316ef04/2023/2023-12-25/IMG_1219.MOV'] Unable to send file: Error`

volumes:
  - ${UPLOAD_LOCATION}:/usr/src/app/upload

UPLOAD_LOCATION:/srv/mergerfs/mergefs/PHOTOS

[uneven oriole]

@summer rain uploaded

@summer rain uploaded

@summer rain uploaded

[summer rain]

I'm thinking my best way forward is a redeploy of some kind.
Is there a process for that where I can keep my users and library folders, but redo everything else?

[uneven oriole]

Successfully submitted, a tag has been added to inform contributors. :white_check_mark:

[warped scarab]

whenever there is no such file or directory log, it means the mount is not properly mounted in the container

[summer rain]

Hi Alex 🙂

I have what appears to be a fully functioning immich deployment though. Just loads of these errors everywhere after the underlying disk was fixed as per original post.

No paths or mounts were changed, only one disk out of 10 was removed and added back into the mergerfs pool. The container and host volumes have remained the same.

From the immich container perspective - it would have just lost about 5% of files (random) in the following directories: /library, /upload, /encoded-video, and /thumbs. These files returned to their original location after around 24h.

Lots of these in microservices too.

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x58270190180] moov atom not found
**upload/library/**0c5a6997-7433-424f-a12e-189f5316ef04/2017/2017-09-14/IMG_2080.MOV: Invalid data found when processing input

This file exists here:
"\\OPENMEDIAVAULT\photos\library\0c5a6997-7433-424f-a12e-189f5316ef04\2017\2017-09-14\IMG_2080.MOV"

Which does match my mapping;
${UPLOAD_LOCATION}:/usr/src/app/upload
/srv/mergerfs/mergefs/PHOTOS

immich-server container path showing file present:
xxx@xxx:/usr/src/app/upload/library/0c5a6997-7433-424f-a12e-189f5316ef04/2017/2017-09-14# ls *2080* IMG_2080.MOV

Server path showing file present:
xxx@xxxx:/srv/mergerfs/mergefs/PHOTOS/library/0c5a6997-7433-424f-a12e-189f5316ef04/2017/2017-09-14# ls *2080* IMG_2080.MOV

I'm more inclined to want a fresh start solution over fixing this - but if I can help solve a potential bug happy to try anything.

[warped scarab]

then that is indeed strange 🤔

[summer rain]

Can I delete the /thumbs directory content and then just run the thumbs job to recreate?
The majority of errors point to /upload/thumbs for both my users.

Running the thumbnail job for ALL didn't resolve, but maybe that will.

TEST:

Error: ENOENT: no such file or directory, access 'upload/thumbs/0c5a6997-7433-424f-a12e-189f5316ef04/09/b5/09b5a804-c4f2-4eae-bc2f-42bf2e291ed7.jpeg'

Went into directory upload/thumbs/0c5a6997-7433-424f-a12e-189f5316ef04/09/b5/ and removed 09b5a804-c4f2-4eae-bc2f-42bf2e291ed7.jpeg.
Ran thumbnail job - nothing found, no tasks.
Put files back.
Completed above with a thumbnail that does not have this problem. Same result. Did not reproduce the issue.

I'm guessing the DB is involved in this. I don't know how the app handles missing thumbs/photos/transcodes.

[warped scarab]

You can run regenerate thumbnail for all

that should take care of all thumbnails

[summer rain]

Already tried, it fixed the thumbnails that were missing visually in the app, but not the errors detailed above. I will re-run ALL again now.

If this fails to resolve the issue, can I delete everything in /thumbs before attempting again? (~100k images)

EDIT: I rebuilt.