#Assets lost during migration

1 messages · Page 1 of 1 (latest)

deep mauve
#

Dear all

I am finally migrating to immich and I am facing an issue with missing assets.

The first attempts were more successful in terms of completeness, but I was facing issues with the timeline as copying photos (from Synology Photos folder) changed timestamps. Now, several thousand photos seem to disappear - this situation seems to be reproducable as I have already retried it.

What I did:

  • Mount USB drive on NAS1
  • Create a tar archive with tar -cf <Synology Photos folder> to an USP drive
  • Unarchive with tar -xf
  • Delete all @eaDir folders with thumbnails recursively
  • Mount USB drive to NAS2
  • Upload photos with the following command:

docker run -it -v "$(pwd)":/import:ro \

-e IMMICH_INSTANCE_URL=http://<host>:2283/api
-e IMMICH_API_KEY=<key>
ghcr.io/immich-app/immich-cli:latest upload . --recursive

Crawling for assets...
Hashing files | ████████████████████████████████████████ | 100% | ETA: 0s | 11929/11929 assets
Checking for duplicates | ████████████████████████████████████████ | 100% | ETA: 0s | 11929/11929 assets
Found 11929 new files and 0 duplicates
Uploading assets | ████████████████████████████████████████ | 99% | ETA: 1s | 31.2 GB/31.2 GB
Successfully uploaded 11910 new assets (31.2 GB)
Skipped 19 duplicate assets (6.6 MB)

Why does the progress bar stay at 99%?

In the Server statistics overview I now see:
Photos: 8.725 (16 GiB)
Videos: 676 (13 GiB)
Usage: 29 GiB

I expected to see 11910 assets in total, but apparently there are only 9401.

Is this a known error or am I doing something wrong? My wife has even 4 times number of photos.

Thank you
Patric

cursive lotusBOT
#

:wave: Hey @deep mauve,

Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich immich.

References

#

Checklist

I have...

  1. :ballot_box_with_check: verified I'm on the latest release(note that mobile app releases may take some time).
  2. :ballot_box_with_check: read applicable release notes.
  3. :ballot_box_with_check: reviewed the FAQs for known issues.
  4. :ballot_box_with_check: reviewed Github for known issues.
  5. :ballot_box_with_check: tried accessing Immich via local ip (without a custom reverse proxy).
  6. :ballot_box_with_check: uploaded the relevant information (see below).
  7. :ballot_box_with_check: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable

(an item can be marked as "complete" by reacting with the appropriate number)

Information

In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:

  • Your docker-compose.yml and .env files.
  • Logs from all the containers and their status (see above).
  • All the troubleshooting steps you've tried so far.
  • Any recent changes you've made to Immich or your system.
  • Details about your system (both software/OS and hardware).
  • Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
  • The version of the Immich server, mobile app, and other relevant pieces.
  • Any other information that you think might be relevant.

Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)

If this ticket can be closed you can use the /close command, and re-open it later if needed.

deep mauve
cursive lotusBOT
stoic vortex
deep mauve
stoic vortex
#

can you execute this? the server may be still processing and is not showing the full stats

docker exec immich_postgres psql --dbname=immich --username=immich -c "select count(*) from assets where status = 'active';"
deep mauve
stoic vortex
#

so it match

#

go to /admin/jobs-status in your server and check if there's any job running

#

give it some time to complete

deep mauve
stoic vortex
#

ok, let's wait for them to finish. the assets are stored in the DB, you can check the number of files under upload and library and compare if they match.
once all the jobs are done, if the stats are still wrong (assuming fs comparison is good), may be a bug on how it's calculated

deep mauve
#

thank you @stoic vortex I will share an update. Is there anything regarding "assuming fs comparison is good" I can do?

stoic vortex
#

count the number of assets in upload and library (if you arr using storage template), they should be 11910 if everything has been uploaded successfully

deep mauve
stoic vortex
#

what's the file count in /volume1/docker/immich/upload/upload

stoic vortex
#

you are the only user using immich, right?

deep mauve
worldly forge
#

Do live photos count as 1 or 2 assets? Perhaps this is the difference?

stoic vortex
#

they count as 2 in the fs, there could be some sidecar files too

worldly forge
#

Right, but how would that count within immich? I'm not at an instance to test, but it could be a possible place for the number mismatch

stoic vortex
worldly forge
#

(for example, if you upload a live photo, does video and photo count increase, or just 1?)

stoic vortex
#

I don't have live photos, so I can't test 😂

worldly forge
#

Or if there's a way to search for live photos then op could see if that's the number difference

deep mauve
#

I used live photos in the past but I do not know how they are persisted. Which file name suffix is that? Is there always a non-live photo created as well with the same name and a different suffix? If that is the case, I could delete my user again and upload only these 2 files, if that helps.

worldly forge
#

Creating a new user and just uploading one live photo would work too

deep mauve
deep mauve
#

I ran several migration tests. In PGAdmin "select count(*) from assets where type = 'VIDEO' and "ownerId" = 'ffe84ecf-b7a0-4c2c-87d0-e802158268ce';" returns 3185 but in the server-status page I see 188 videos. How is that possible? And the strange thing is, short after immich-go process I saw 2828 videos and then this number decreased - maybe it was even 3185 immediately after upload. Since you mentioned live photos, I used immich-go option "--manage-heic-jpeg=NoStack" (I hope my interpretation of this option was correct). The import logs do not show errors except warnings on "warning=can't read metadata: EOF". When I press CTRL+A in Photos search I only see 188. Why are all others hidden?

deep mauve
stoic vortex
deep mauve
stoic vortex
#

is up to you to investigate, since I have no way to know what those assets are

deep mauve
stoic vortex
#

why would you want to unhide them? they are hidden for a reason, you have to analyze what assets are hidden and find out why. maybe they don't really exists in the FS. maybe they are part of a stack