#Not reading all GPS data from a large collection of files

1 messages · Page 1 of 1 (latest)

finite stag
#

Have a library of ~180,000 files, a mix of JPG/Sony ARW/Canon CR2/CR3. Conservatively, around 80% are tagged: for JPGs, the GPS data is inside the files, whereas for ARW/CR[2/3] the GPS coordinates are in XMP sidecars. On importing the entire library, the GPS coordinates are never completely read: Immich reads anywhere between a couple of thousand and around 50-90K files. Never the whole set. I have tried:

  • running Extract Metadata on Missing, that completes instantly (so it clearly thinks it's read all the sidecars)
  • Discover Sidecar Metadata and then Extract Metadata on All

It never seems to quite go beyond half 'ish of the entire metadata set. What could I investigate to try to get it to 100%?

hallow tokenBOT
#

:wave: Hey @finite stag,

Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich immich.

References

#

Checklist

I have...

  1. :blue_square: verified I'm on the latest release(note that mobile app releases may take some time).
  2. :blue_square: read applicable release notes.
  3. :blue_square: reviewed the FAQs for known issues.
  4. :blue_square: reviewed Github for known issues.
  5. :blue_square: tried accessing Immich via local ip (without a custom reverse proxy).
  6. :blue_square: uploaded the relevant information (see below).
  7. :blue_square: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable

(an item can be marked as "complete" by reacting with the appropriate number)

Information

In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:

  • Your docker-compose.yml and .env files.
  • Logs from all the containers and their status (see above).
  • All the troubleshooting steps you've tried so far.
  • Any recent changes you've made to Immich or your system.
  • Details about your system (both software/OS and hardware).
  • Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
  • The version of the Immich server, mobile app, and other relevant pieces.
  • Any other information that you think might be relevant.

Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)

If this ticket can be closed you can use the /close command, and re-open it later if needed.

slate pawn
#

On importing the entire library, the GPS coordinates are never completely read
Have you tried a small import batch?
How do you perform import?
What are job concurrency settings - default or increased?
What could I investigate
Enable Verbose logging in the server settings. Choose a JPG with geolocation metadata that is not reflected in Immich. Trigger "Refresh metadata" for that image. Check / post server logs.

finite stag
#

Thanks. In order:

  1. I have not tried smaller batches: since my entire photo library is two folders with a few hundred subfolders (by date, and two just because no space on one drive, so I split across two drives), I just added the top level folder. Is it possible to run a import on a part of a bigger folder, or do I need to manually add a sub-subfolder?
  2. External Libraries > Folders > Add, added both huge folders.
  3. Concurrency is at 5, so default.

For investigation: done, enabled verbose logging. Here is what I got:

📎 message.txt
slate pawn
hallow tokenBOT
slate pawn
#

Could you tell about your setup? Hardware, OS, docker, ...

finite stag
#

Yep. Running on a stack spun up by portainer that sits on a ESXi Ubuntu 22.04. Docker host is allocated 4 cores of a Ryzen 2600, and 20GB of RAM.

slate pawn
hallow tokenBOT
slate pawn