Hashing numbers don’t match

[fast cipher]

Im opening a new issue here to keep it localized.

I installed both immich app and web anew fresh
No items have been synced yet, I’ve let the beta timeline hashing complete first

This is what I see now and it’s not matching the found items.

1. found items are wrong (but it’s been mentioned that this is actually expected as it counts all items including shared albums). I’ve ~9k images, it found ~14k.
2. hashed items are off by a few, yet it states to be finished with the job. The discrepancy matches exactly the count of items in „recently viewed“ of the iPhone photos app. Not sure if that’s coincidence.
3. it states „memories 0“, but that’s wrong.

Before I proceed with actual sync now I’d like to know what exactly is causing it and if it’ll cause issues (well, I’m sure it’ll: it will try to sync either number of items and thus either never finish sync or oversync, one of both)

All systems are up to date on latest version.

[heady tiger]

:wave: Hey @fast cipher,

Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich .

References

• Container Logs: docker compose logs docs
• Container Status: docker ps -a docs
• Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
• Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA

Checklist

I have...

1. :ballot_box_with_check: verified I'm on the latest release(note that mobile app releases may take some time).
2. :ballot_box_with_check: read applicable release notes.
3. :ballot_box_with_check: reviewed the FAQs for known issues.
4. :ballot_box_with_check: reviewed Github for known issues.
5. :ballot_box_with_check: tried accessing Immich via local ip (without a custom reverse proxy).
6. :blue_square: uploaded the relevant information (see below).
7. :ballot_box_with_check: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable

(an item can be marked as "complete" by reacting with the appropriate number)

Information

In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:

• Your docker-compose.yml and .env files.
• Logs from all the containers and their status (see above).
• All the troubleshooting steps you've tried so far.
• Any recent changes you've made to Immich or your system.
• Details about your system (both software/OS and hardware).
• Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
• The version of the Immich server, mobile app, and other relevant pieces.
• Any other information that you think might be relevant.

Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)

If this ticket can be closed you can use the /close command, and re-open it later if needed.

[heady tiger]

Successfully submitted, a tag has been added to inform contributors. :white_check_mark:

[wintry belfry]

my gut feeling is again, related to Shared Albums icloud assets, those mismatches could be assets that are no longer accessible but are stil presented in the iCloud database 🤔

[fast cipher]

Is there any way I can find these assets it tried to hash, so I can check in iCloud?

This is a rather important part I think because if this is off, all other will be too I assume

[wintry belfry]

You can export the sqlite database and send it to me. I can help look for them

The option to export the database is in the beta stats sync page

[fast cipher]

Sent in pm

[wintry belfry]

Hey can you also send the latest mobile app logs?

[fast cipher]

Will do as soon i moved these out

[fast cipher]

1. The discrepancy between reported local files and hashed files finally is cleared: _I had unsupported files like .mod and .mts in the photo library, which somehow where possible to add in past, no more now...
2. The new mobile app logs are sent in PM

[fast cipher]

Unfortunately, the import did not complete, and hash numbers are now again out of sync (after letting sync run overnight)
@wintry belfry do you want me to open a new thread to go through this or proceed here?

Basically the behaviour was:

• yesterday hashing completed as we know and numbers of local assets > hashed assets where identical

• I selected the album to sync (Recents (All))

• It started syncing and in the "View Details" I could see how things progressed

• I let it run (open screen, active app, phone set to never turn off) overnight and in the morning:

• Hashed assets went down in numbers but are now increasing again slowly approaching the local assets number. I added NO new assets locally in this time

• Sync stopped, in the morning when I came in there was no item in the "view details" at all, and I closed the app, reopened, which did not bring back any items in "view details"... until suddenly it started syncing again. I have unfortunately no way of telling when it stopped, but it stopped early because after I started sync it went relatively quickly to about 1500 assets, and now it is on 1768 synced assets. So most of the night it idled. And when I look at the details now, I see tons of items... but none are actually progressing, they are all on 0--MB/s

• The "Enable backup" button thing said nothing when i woke up but now again says "queuing 99/7578"

THere was no special action on my end other than wait with the app open and suddenly that process started again

I think there are at least 2 or 2 main isues here:

• why does hashed numbers go down again after they where 1:1?
• why does the sync hang up? How to "retrigger" it without having to wait an undefinte amount of time once it hangs up?

[wintry belfry]

Can you share the screenshot again?

[fast cipher]

Just noticed the counts of "backed up photos and videos" is also inequal to "remote assets" and all of these are inequal to web-interface stats
first says 1768, second says 1957, third says 1775

I am going to scratch this again, this ain't working with large libraries.

I think there was a thing to do an initial sync, a script or so? Is that immich-go?

[wintry belfry]

My hunch is this is related to iCloud shenanigan, will need more investigating

[wintry belfry]

the web stats might have included live photos

or vice versa

[fast cipher]

In any case the queue had all files on 0mb, something clearly wrong

I scratched, not worth waiting for long time on this one

about that tool to do initial sync of large libs, is that something I dreamt up? or simply not the recommended way?

[wintry belfry]

I will need to do more investigation, I have pretty large library but all my assets are locally on my phone

and it churns through very quickly

so I need to recreate a large iCloud library to figure out what exactly going on here

Thanks for helping with testing and reporting

[fast cipher]

I cant put all my photos onto my phone, it blows the memory
I do have them all on my mac thou, so that is why I am asking if there are other ways

[wintry belfry]

@hollow grail Do you have any hunch on why the hashed assets number going down?

[wintry belfry]

yeah the upload will download, then upload then discard the downloaded files from iCloud

[fast cipher]

Clearly with the phone this one wont do, be it because it has to pull from icloud a few assets, or my internet not good, or what it be...
I think if I sync the initial thing without the app, but I am not sure if that immich-go is the right way?

[wintry belfry]

You can use icloudp to get all the assets from iCloud and then upload them to the app via the web/CLI or immich-go

[fast cipher]

iCloudPD?

[wintry belfry]

yes

[whole shore]

icloudpd

[fast cipher]

iCloudPD: ... it downloads from iCloud even if I already have them on the mac
Apple Photos export (wow, did not even know it exists!): ... useless. Crashes at a few pics
osxphotos (python lib): ... guess what, actually capable and reports... 14641 found items, exported 11445 items, missing 4828 (shared albums?). Elapsed time: 0:01:14

Analyzing those missing items, I can see them locally in my mac in the photos library folder and also in photos app. For some reason, they are labelled as missing thou and are not exported.
According GPT this is because it is a shared folder and thus while "visible" ... not in the main database (even if it is MY foto in MY shared album)
I think... Photos app is massively more buggy than anything else I ever saw.

Basically, you are either locked in or you ... well, better do not ever start using it becuase you may just not be able to actually "own" all that you put into there. Not surprised, but it is annoying.

[hollow grail]

I just remembered the weird thing were the updated time of the local assets were changed by iOS when we were just swiping through them in the Photos app. And I think we reset the checksum when the updated time is changed. Let me test this real quick. If that's the case, this would mean that the app is rehashing few assets every-time you open it after using the photos app, which could be a problem on the older hardware

[hollow grail]

I believe your device just amplifies the existing issues with our implementation, which is a good thing because we can finally see and try to address them. Hopefully, you should be able to use the Immich app to backup your library soon-ish 😅

[fast cipher]

Oh comon now...
After the osxphotos esport, I can see all pics in the folder when I open them in mac
However, on the immich web intereface, about 80% of those photos are shown like in the screenshot when clicking upload and selecting the folder where they are in
And uploading them does nothing

When the "file icons" actually show a "preview" then it uploads them
yet the photos are intact, thy are visible on my mac (see second screnshot)

🤣

[wintry belfry]

I have a secret to share

you can actually download the Immich iOS app and run it on Mac

[fast cipher]

Right, I did that in past

[wintry belfry]

then you can select the folder to upload photos from your Mac to Immich

damn

not a secret anymore

[fast cipher]

but that then duplicates all images, as soon as you install it on iphone

[wintry belfry]

hmm

I guess photos on Mac is not the original version then

[fast cipher]

Not sure, but I asked in past and you said it is intended to NOT be installed on more than one device

As in, it was expected that to happen

[wintry belfry]

we never fully tested it

so looks like it doesn't work

[fast cipher]

I did that in past, yes, and it basically duplicated everything. I asked here in discord and you suggested to use only one instance (there should be still a thread I think)

[wintry belfry]

so Photos you have on Mac are not physically on the device then, probably still on iCloud

[fast cipher]

No, they are

Fully on my device - on mac
Not on iPhone, no place there

[wintry belfry]

I see

alright scratch that idea

[fast cipher]

I have the data ready. Just that now #1407378678630973550 message happens
lol

[fast cipher]

I am closing this, although there clearly is room to improve this... this aint a github ticket and i assume it wont be used as "workable task"
If need of course re-open