#Deduplication after uploading via CLI

1 messages · Page 1 of 1 (latest)

zealous beacon
#

Hi everyone, I’m a new user of Immich. I have it running in my UGreen NAS with a docker container.

I had 500+ GB of images to upload from iCloud. Since I had everying exported from iCloud, I used CLI to upload to Immich. All good, everything is now uploaded.

Finally I thought I could enable my iPhone backup (recents and albums - which is a duplicate of what I had already uploaded via CLI) but as soon as I enabled, I started seeing duplicates of the same image. Quickly I paused the syncing.

With some reading I found out that, it’s because the CLI uses a checksum and uploading via iPhone uses a UUID to identify the duplicates and since it’s two different things, identifying the duplicates is impossible.

Is my understanding correct? Or this has any solutions that exist already? Any help is appreciated. Thanks in advance.

tribal moatBOT
#

:wave: Hey @zealous beacon,

Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich immich.

References

#

Checklist

I have...

  1. :blue_square: verified I'm on the latest release(note that mobile app releases may take some time).
  2. :blue_square: read applicable release notes.
  3. :blue_square: reviewed the FAQs for known issues.
  4. :blue_square: reviewed Github for known issues.
  5. :blue_square: tried accessing Immich via local ip (without a custom reverse proxy).
  6. :blue_square: uploaded the relevant information (see below).
  7. :blue_square: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable

(an item can be marked as "complete" by reacting with the appropriate number)

Information

In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:

  • Your docker-compose.yml and .env files.
  • Logs from all the containers and their status (see above).
  • All the troubleshooting steps you've tried so far.
  • Any recent changes you've made to Immich or your system.
  • Details about your system (both software/OS and hardware).
  • Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
  • The version of the Immich server, mobile app, and other relevant pieces.
  • Any other information that you think might be relevant.

Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)

If this ticket can be closed you can use the /close command, and re-open it later if needed.

thorn acorn
#

The phone uses checksums too; if duplicates are being uploaded that means the files are not precisely identical

zealous beacon
#

I can confirm it’s the same image, with same name, size and resolution.

thorn acorn
#

Those aren’t really sufficient indicators, a single bit change would be enough to change the checksum

left tartan
#

iOS also does some funky things with what media is actually presented when a checksum is done

There is an existing FR on GitHub to have some properly blocking of re-uploading assets from iOS to Immich. I ended up using a Shortcut on my phone to put images into a dedicated album to not be uploaded