Storage Template Migration Changing Filename | Immich | Page 1

stoic nimbus Aug 24, 2024, 9:41 AM

#

It seems like running storage template migration is changing my file names, which is unexpected. It seems to only happen for videos. For example a diff of before and after migration:

-    /data/library/admin/2020/2020-03-17/IMG_0219(1).MP4
-    /data/library/admin/2020/2020-03-17/IMG_0219(1).MP4.xmp
+    /data/library/admin/2020/2020-03-17/IMG_0219.MP4
+    /data/library/admin/2020/2020-03-17/IMG_0219.MP4.xmp

-    /data/library/admin/2021/2021-07-25/IMG_2218-edited.MP4
-    /data/library/admin/2021/2021-07-25/IMG_2218-edited.MP4.xmp
+    /data/library/admin/2021/2021-07-25/IMG_2218.MP4
+    /data/library/admin/2021/2021-07-25/IMG_2218.MP4.xmp

I was wondering if this is expected behavior? It doesn't seem to happen for photos, which is why I'm confused.

meager topazBOT Aug 24, 2024, 9:41 AM

#

:wave: Hey @stoic nimbus,

Thanks for reaching out to us. Please follow the recommended actions below; this will help us be more effective in our support effort and leave more time for building Immich immich .

References

Container Logs: docker compose logs docs
Container Status: docker compose ps docs
Reverse Proxy: https://immich.app/docs/administration/reverse-proxy

Checklist

:ballot_box_with_check: I have verified I'm on the latest release(note that mobile app releases may take some time).
:ballot_box_with_check: I have read applicable release notes.
:ballot_box_with_check: I have reviewed the FAQs for known issues.
:ballot_box_with_check: I have reviewed Github for known issues.
:ballot_box_with_check: I have tried accessing Immich via local ip (without a custom reverse proxy).
:ballot_box_with_check: I have uploaded the relevant logs, docker compose, and .env files, making sure to use code formatting.
:ballot_box_with_check: I have tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable

(an item can be marked as "complete" by reacting with the appropriate number)

If this ticket can be closed you can use the /close command, and re-open it later if needed.

meager topazBOT Aug 24, 2024, 9:42 AM

#

meager topaz :wave: Hey <@131918669197475840>, Thanks for reaching out to us. Please follow ...

Successfully submitted, a tag has been added to inform contributors. :white_check_mark:

wind idol Aug 24, 2024, 2:08 PM

#

Do you know which operations you have done on those files to have the xmp file?

stoic nimbus Aug 24, 2024, 8:20 PM

#

wind idol Do you know which operations you have done on those files to have the xmp file?

Hey, I’m not sure. Would checking the XMP file tell me?

wind idol Aug 24, 2024, 8:21 PM

#

probably

stoic nimbus Aug 24, 2024, 8:41 PM

#

The 2019 file XMP:

<?xpacket begin='' id='W5M0MpCehiHzreSzNTczkc9d'?>
<x:xmpmeta xmlns:x='adobe:ns:meta/' x:xmptk='Image::ExifTool 12.78'>
<rdf:RDF xmlns:rdf='http://www.w3.org/1999/02/22-rdf-syntax-ns#'>

 <rdf:Description rdf:about=''
  xmlns:exif='http://ns.adobe.com/exif/1.0/'>
  <exif:GPSLatitude>REDACT</exif:GPSLatitude>
  <exif:GPSLongitude>REDACT</exif:GPSLongitude>
 </rdf:Description>
</rdf:RDF>
</x:xmpmeta>
<?xpacket end='w'?>

#

Do you think this is related to the issue?

#

Took a look at the code and don't see any reasons for why this would be happening.

wind idol Aug 24, 2024, 8:43 PM

#

this is an Upload Library, correct?

stoic nimbus Aug 24, 2024, 8:44 PM

#

yes

wind idol Aug 24, 2024, 8:47 PM

#

maybe some bug 🤔, can you describe the step you take to reproduce this issue?\

stoic nimbus Aug 24, 2024, 8:51 PM

#

Yea let me see if I can repro it. Is there a good way to do so? I can't upload identical files I think.

#

ok I have no idea, I can't get it to happen anymore

wind idol Aug 24, 2024, 9:01 PM

#

if you happen to be able to reproduce it, can you give me an update?

stoic nimbus Aug 24, 2024, 9:01 PM

#

Will do

wind idol Aug 24, 2024, 9:01 PM

#

Thanks man

stoic nimbus Aug 24, 2024, 9:13 PM

#

No problem, thank you for the quick triage!

#

While I have you, I had another issue where a lot of my live photos were renamed from .MOV to .MOV.mov in a recent storage template migration as well. It's yet another one I can't reproduce, but is there a way I can rename them without reuploading them?

#

Would i have to actually modify the database, or is there an easier way?

pale turret Aug 24, 2024, 9:26 PM

#

You can always re-run the template migration (maybe after changing the template and/or fixing the code), it will just move files that don't have the proper path based on the new template. Best case this will fix things, worst case it will leave everything where it is (or add another .mov if the code really is weird)

stoic nimbus Aug 24, 2024, 9:27 PM

#

I'm just worried that it would lose track of the files since the naming wouldn't match the database. Have a decent number of files this happened to but no idea how as I cannot repro

#

find . -type f -name "*.MOV.mov" | wc -l
677

#

It looks like I can just batch rename all of them though, then do a check all -> repair all and that'll do it?

pale turret Aug 24, 2024, 9:28 PM

#

The migration job updates the database. The resulting paths might be wrong/weird, but at least Immich still is able to find the files.

#

Don't change the names manually! Immich won't be able to find the files (unless you also manually update the DB - which I don't recommend)

stoic nimbus Aug 24, 2024, 9:28 PM

#

Even if it can't find a given file?

pale turret Aug 24, 2024, 9:28 PM

#

Let's focus on fixing the code. What's your storage template?

#

If it can't find a file without you doing manual stuff, that's surely a bug. And yes, it might help to manually revert to the name known in the database. No guarantees. Be careful.

stoic nimbus Aug 24, 2024, 9:29 PM

#

{{y}}/{{y}}-{{MM}}-{{dd}}/{{filename}}

The thing is, I don't know if the code is broken anymore as I've tried uploading live files with a .MOV extension and then running the template, and it works just fine now.

#

So I mostly just want to fix my existing library which now has a bunch of files with .MOV.mov extensions. You're saying I can do this with the migration job somehow?

pale turret Aug 24, 2024, 9:31 PM

#

On the CLI, you can try "mmv": mmv "*.MOV.mov" "#1.mov"

stoic nimbus Aug 24, 2024, 9:31 PM

#

I ran the storage migration last night when this happened, and I can't get it to happen again. Really weird. These were files I uploaded a long time ago so maybe something funny with the database?

pale turret Aug 24, 2024, 9:31 PM

#

But you need to be careful to end up with a path that Immich has in its assets database table

#

I'm heading out now. Let's look into this in more detail later.

stoic nimbus Aug 24, 2024, 9:32 PM

#

Thanks, appreciate it.

#

In theory all of these paths should already be in the DB table since they exist in current storage template paths.

#

Ok so I resolved the library issue by just doing a batch rename with find, then did a check all and repair. The check all matched the number of files that find moved so looks good.

Happy to keep debugging this later, hopefully we can root cause it haha

stoic nimbus Aug 24, 2024, 10:00 PM

#

Not urgent, did another zfs diff after doing this process. Everything looks good, most of the diff was just .MOV.mov changes, but noticed a few entries like

Binary files /mnt/cache/photos/library/.zfs/snapshot/manual-2024-08-24_14-04/admin/2017/2017-11-16/IMG_20171116_130340+1.jpg and /mnt/cache/photos/library/admin/2017/2017-11-16/IMG_20171116_130340+1.jpg differ
Binary files /mnt/cache/photos/library/.zfs/snapshot/manual-2024-08-24_14-04/admin/2017/2017-11-16/IMG_20171116_130340+2.jpg and /mnt/cache/photos/library/admin/2017/2017-11-16/IMG_20171116_130340+2.jpg differ
Binary files /mnt/cache/photos/library/.zfs/snapshot/manual-2024-08-24_14-04/admin/2019/2019-01-20/IMG_20190120_162917-EFFECTS+1.jpg and /mnt/cache/photos/library/admin/2019/2019-01-20/IMG_20190120_162917-EFFECTS+1.jpg differ

Seems a bit strange, I saw in the storage template code that it appends a +1 if a duplicate occurs, but I don't get why the file contents would change.

pale turret Aug 25, 2024, 9:04 AM

#

Maybe ...40+1 and ...40+2 were swapped?

stoic nimbus Aug 25, 2024, 9:21 AM

#

[~]$ openssl md5 /mnt/cache/photos/library/.zfs/snapshot/manual-2024-08-24_14-04/admin/2017/2017-02-22/*
MD5(/mnt/cache/photos/library/.zfs/snapshot/manual-2024-08-24_14-04/admin/2017/2017-02-22/IMG_20170222_135958+1.jpg)= c22ab816a42ff43bc8e2ffb10e3b5fc7
MD5(/mnt/cache/photos/library/.zfs/snapshot/manual-2024-08-24_14-04/admin/2017/2017-02-22/IMG_20170222_135958+1.jpg.xmp)= 9271b99a903dfd080c7a44d61babf280
MD5(/mnt/cache/photos/library/.zfs/snapshot/manual-2024-08-24_14-04/admin/2017/2017-02-22/IMG_20170222_135958+2.jpg)= 356bc2fa57c031bd4cf2d167947472e0
MD5(/mnt/cache/photos/library/.zfs/snapshot/manual-2024-08-24_14-04/admin/2017/2017-02-22/IMG_20170222_135958+2.jpg.xmp)= 9271b99a903dfd080c7a44d61babf280
MD5(/mnt/cache/photos/library/.zfs/snapshot/manual-2024-08-24_14-04/admin/2017/2017-02-22/IMG_20170222_135958+3.jpg)= dc1794ec8e1e1e68ef80616c8877706f
MD5(/mnt/cache/photos/library/.zfs/snapshot/manual-2024-08-24_14-04/admin/2017/2017-02-22/IMG_20170222_135958.jpg)= a086d07f8a7255459727cfd478d37ea4
[~]$ openssl md5 /mnt/cache/photos/library/admin/2017/2017-02-22/*
MD5(/mnt/cache/photos/library/admin/2017/2017-02-22/IMG_20170222_135958+1.jpg)= dc1794ec8e1e1e68ef80616c8877706f
MD5(/mnt/cache/photos/library/admin/2017/2017-02-22/IMG_20170222_135958+2.jpg)= c22ab816a42ff43bc8e2ffb10e3b5fc7
MD5(/mnt/cache/photos/library/admin/2017/2017-02-22/IMG_20170222_135958+2.jpg.xmp)= 9271b99a903dfd080c7a44d61babf280
MD5(/mnt/cache/photos/library/admin/2017/2017-02-22/IMG_20170222_135958+3.jpg)= 356bc2fa57c031bd4cf2d167947472e0
MD5(/mnt/cache/photos/library/admin/2017/2017-02-22/IMG_20170222_135958+3.jpg.xmp)= 9271b99a903dfd080c7a44d61babf280
MD5(/mnt/cache/photos/library/admin/2017/2017-02-22/IMG_20170222_135958.jpg)= a086d07f8a7255459727cfd478d37ea4

#

Nice guess lol

#Storage Template Migration Changing Filename

References

Checklist