Identify photo date by age with artificial intelligence

[quaint raptor]

I would be happy to know what you think about this idea and whether it is possible to implement it in Immich
https://github.com/immich-app/immich/discussions/5133

in short:

[covert barn]

Wait so it does facial recognition on a person in the photo, and if it knows their birthday it sets the date of the photo based on their perceived age?

Technically possible, but seems very error prone

[frail kestrel]

It would be pretty cool, but yeah age detection via ml is not very accurate

[quaint raptor]

@covert barn @frail kestrel
In fact, there is a very advanced model and its level of accuracy is incredibly high
Take a look here:
https://huggingface.co/spaces/iitolstykh/age_gender_estimation_demo

[frail kestrel]

Cc @inland torrent

[covert barn]

It was off for me by 5 years
I mean that's not terrible, but I dunno if photos being dated with that kind of error is super great

[quaint raptor]

Did you use the default model parameters in huggingface?

[covert barn]

yes

are there better ones?

[quaint raptor]

In fact it depends terribly, according to their official article there are very high accuracy ratios according to certain ages as I read
This can vary depending on the image and the type of model used (face only or face + body) to analyze age

Regarding your question about changing parameters related to the accuracy of the model in huggingface, I am actually still researching about it

[covert barn]

So it gets a mean absolute error of 6.66 on ideal body photos

and vary's but is generally above 5

again, not terrible
But not exactly precise

[quaint raptor]

However, it is more accurate in humans in certain scenarios
And according to the graph at young ages it was found to be more accurate compared to humans and the probability is that most of the pictures without the EXIF are when we were little

[quaint raptor]

I'm thinking of trying to start work on it and then do a PR
I'm currently in college so my code is not a hit,
I wanted to know if this proposal is something that will likely be accepted by the Immich team so that I know whether to start working on trying to apply it

Or should I wait more for Mertalev's response

[frail kestrel]

I'll pull in all the <@&980972470964215870> s to get their thoughts on this

[quaint raptor]

Thank you!
I'll wait, I just want to point out that more information appears on Github besides the sketch

[wheat gorge]

My thoughts are this isn't going to be particularly accurate in a lot of cases, and I also don't really see the benefit of this being in Immich overall. We've long talked about eventually making it possible to add plugins to Immich (although this is a long way in the future) and I think this would work well for this kind of thing, but personally I don't see it being worthwhile bringing this to core Immich

[thin fjord]

Mixed feelings. I don't feel like it's really needed but it would be a cool plugin if Immich could have one. I don't see how it helps you organize your assets

[ionic shale]

didn't someone recently add an age/birthday field to faces, or was that just a PR?

[thin fjord]

Yes it's already implemented

[quaint raptor]

@thin fjord @wheat gorge @ionic shale

Thanks for your insightful response regarding the potential integration of age recognition features into Immich. I truly appreciate your consideration and wanted to offer further information that might illuminate the significance and practicality of incorporating the MiVOLO model.

Enhanced Accuracy: MiVOLO stands out as a high-performing model, particularly in age and gender estimation. Compared to existing models, MiVOLO's precision and accuracy surpass many contemporary alternatives. Its state-of-the-art capabilities provide a level of accuracy essential for accurately arranging photos in chronological order.

User Experience Improvement: Immich's strength lies in curating memories through photos. Integrating MiVOLO’s age recognition feature would significantly enhance the user experience. By organizing photos according to age, Immich will offer users a more intuitive and emotionally resonant journey through their visual memories.

Future-Proofing Immich: While the prospect of plugin capabilities is on the horizon, the immediacy of improving chronological arrangement warrants consideration. Integrating MiVOLO now aligns with Immich's ethos of continuously enhancing its functionalities for user satisfaction.

Competitive Edge for future contributors: In an era where user-centric features distinguish platforms, Immich can gain a competitive edge by being one of the pioneers to incorporate accurate age recognition into its service.

Additionally, the ability to test the MiVOLO model through platforms like Hugging Face allows for hands-on exploration, showcasing its potential in a user-friendly manner.

I firmly believe that integrating the MiVOLO model into Immich will not only address the chronological organization issue but also elevate the platform’s standing among users. I am willing to collaborate further on this matter and provide any necessary support to facilitate its integration.

Thanks

[lucid idol]

is that from ChatGPT?

[frail kestrel]

Suuure looks like it

[quaint raptor]

Actually no, I don't speak English so I used GPT to word it better for better understanding because maybe my previous explanation wasn't good or convincing enough

You can be sure I know what I'm talking about, you're not talking to GPT 🙂

In addition, everything I wrote is correct, the model is indeed considered state of the eart in relation to many other models, as you can see
https://github.com/WildChlamydia/MiVOLO#mivolo-multi-input-transformer-for-age-and-gender-estimation

It is also available on huggingface for trial editing if you want to try it out
https://huggingface.co/spaces/iitolstykh/age_gender_estimation_demo

And the claims are claims that I chose to represent

[lucid idol]

Nice!

[wheat gorge]

I do appreciate you taking the time to further explain the issue, however I still believe that this doesn't really belong in core Immich. I'd say that it would be worthwhile making a discussion point on github for this and seeing how it does with votes etc, but I would guess that the amount of people who would actually care to use this feature would be minimal, yet this would be something we need to maintain for the minority of users. When considering what to implement in the Immich core feature base, ongoing maintainence and feature creep are huge factors as at the end of the day we are just a small team of people working on this in our spare time, so every additional feature is another thing we have to support 🙂

[serene condor]

I didn't read the whole convo, but this would be cool to enable a user to do via the api/webhooks/plugin, etc., but agree it's probably more niche than anything.

[quaint raptor]

I'm sorry if it alerts everyone when I post here.
Thank you for your response, I completely understand this matter and still want to say thank you to the team that works day and night to bring us Immich

I want to prepare it even if in the end it is decided not to enter Immich, or yes if, as you said, there will be enough votes for it on Github and the team will agree to the proposal. (Sometimes tangible things work better than written things)

I would like to know if this is all the stuff I need when I build it?

1. Information about the age set in Immich (stored in a database I guess)
2. Does the image have EXIF data or not (I assume this is something I can extract from MetadataExtraction or is it also something stored in a database?)

[lucid idol]

I have the same consensus with Zack that this is niche and it would create additional things to maintenance while not serving a lot of needs

[inland torrent]

I agree with others that this would make for a good plugin, possibly part of a group of toggleable heuristics to help with untagged images. I also agree that it's niche for something to include in the core app. Once you've gone down to:

1. People with a birth date set
2. Images with no creation date metadata
3. People for whom this model would be accurate enough to be useful

You've already filtered out most images in Immich, so the positive impact of it would be limited.

If you do want to work on it, though, this is more or less what it would require:

1. The genderage model is not included in the HF repositories as they are currently unused by the ML service, so this would need to be added to them
2. The face recognition code here would need to be updated to also run the genderage model and add it to the response

• https://github.com/immich-app/immich/blob/c8aa782fef15f8bbad9f0ac1f475929fc51dc7f0/machine-learning/app/models/facial_recognition.py#L55C13-L55C13
• Depending on speed, it may be need to be optional to prevent a slowdown for users who don't use this feature

3. There would need to be a check when processing these faces to see if they're associated with a person who has a birth date set and if the image in question does not have asset metadata

• This would be done here https://github.com/immich-app/immich/blob/c8aa782fef15f8bbad9f0ac1f475929fc51dc7f0/server/src/domain/person/person.service.ts#L239C6-L239C6
• Some edge cases to handle in case the same person appears multiple times in an image and has different inferred ages, or if there are multiple people and the predicted date is inconsistent between them

4. The system config would need to be updated to make this optional since this is a niche feature

• https://github.com/immich-app/immich/blob/c8aa782fef15f8bbad9f0ac1f475929fc51dc7f0/server/src/infra/entities/system-config.entity.ts#L16C18-L16C18
• https://github.com/immich-app/immich/blob/c8aa782fef15f8bbad9f0ac1f475929fc51dc7f0/server/src/domain/system-config/system-config.core.ts#L27C14-L27C14

5. The admin settings for facial recognition would need to include a toggle for it to update the system config

I will personally not be working on this, but I can review a PR that implements the above

[wheat gorge]

I would however suggest waiting to spend the effort until we have some plugin style system in place though, I don't see us merging something like this into the app, and most of the core contributors are in agreement with that. We have to draw a line somewhere with what we accept as in-scope for the core Immich project, sorry it has to be here 🙂

[quaint raptor]

Thank you very much for this, it will definitely be helpful

[quaint raptor]

Absolutely understandable, I'll take it as a personal project for fun 🙂

Thanks for your thoughts on this!

I'm writing this because maybe in the future it will be useful when plugins arrive at Immich
It is possible to verify the answer of the face recognition model with help from Clip when the queries to the model can be
Is the person 1,2,3,4,5,...100 years old?
The Clip model returns the probability of what is written where the value with the highest probability is the one that most likely answers the query

It can help with model accuracy
Although according to what I checked, the Clip model is currently not trained on age detection

[quaint raptor]

Here is an example of a picture from a data set (according to the label of the data set **fairface **the age of the child is between 3-9) according to the Clip guess of
laion/CLIP-ViT-H-14-laion2B-s32B-b79K
This is the results obtained

[frail kestrel]

These classes are steering it fairly heavily, how does it do if you give it 1-99 as options?

[quaint raptor]

Actually it doesn't do well with about that much, so because the age detection model has a deviation of 4 you can give the clip model 4 years per side to verify the answer of the age detection model

[quaint raptor]

@frail kestrel @inland torrent
I wanted to know as soon as I upload a file what the complete process is in terms of the workflow of the code up to the analysis of the image by face (person)

(This is different from the original logic that does face recognition according to an age recognition algorithm)
That is, what is the part that is responsible for associating the image with a person?
And where is the trigger that associates an image with a person?
Is it possible to get an explanation of how to implement this in the code? In terms of names of functions, etc., the code is a bit difficult for me to understand without comments, so I would appreciate an explanation

[inland torrent]

It might be helpful to take a look at how and when the jobs are queued here https://github.com/immich-app/immich/blob/309be88ccd7516f9b9de43593ddb5a29a96c2202/server/src/domain/job/job.service.ts#L161

This is what queues the next jobs after a certain job completes, forming a pipeline. The most relevant jobs there are metadata extraction and facial recognition.

When looking at the implementations, also keep in mind that the server code is split between “domain” and “infra”. Services under domain handle conditional app logic, while repositories under infra handle I/O and are meant to be more generic and reusable.

For example, facial recognition happens in the person service here https://github.com/immich-app/immich/blob/309be88ccd7516f9b9de43593ddb5a29a96c2202/server/src/domain/person/person.service.ts#L213 and as part of that, it calls the machine learning repository to get the actual model outputs.

[quaint raptor]

Thank you very much!

[quaint raptor]

I sat down to figure out how to best implement this I researched Immich's face recognition process and realized that it works without clustering.
These compare the images that are uploaded against the embedded image created from all of the person's images

My thought was to follow the diagram but for this purpose I need clusters to create it in the best way, I would be happy to hear what you think about this way and maybe there is a better and smarter way that I don't know

I made a flow chart

[inland torrent]

This is a ton of complexity to add for a niche feature, and it isn't clear that it would actually be accurate.

In general, I think the premise of age-based photo dating is interesting, but sometimes when you get an interesting idea, you sit down and look at the details and realize it isn't as viable as you'd hoped. I think this is one of those cases, so I would suggest going in a different direction.

[lucid idol]

@quaint raptor you can implement this feature as a fork of your own. I don’t think I will accept this PR into Immich’s core feature following the same reason I and the team have discussed with you. Thank you for discussing this feature with us.

[quaint raptor]

accepted. Thanks for your opinions

[lucid idol]

We understand the pain point! We hope to eventually have a plug-in system where you can build this and use it in your instance

[quaint raptor]

I look forward to this addition, for now I will build it as a side fork.
What I wanted to know in the question I asked is do you have any ideas/libraries that you know so that I can approach the work more easily.

I have now found a library that made development very easy for me without the need for clusters, I will continue to work on it (if anyone from the community would like to work on it together with me, feel free to send a private message)

Thanks for your hard work Alex and team

[stone hare]

What I do to detect the possible date of pictures without right date:
I try to use similarity search (in digiKam) and sometimes I find similar photos having the right date (or a better date).
Maybe this idea can be used when sorting or displaying assets that belong to the same person. They could be clustered by similarity. I don't know if the original cluster is still available after merging faces. Filtering by cluster and combining with dates could be easy and pragmatic.

[quaint raptor]

After thinking I created a completely different way than what was suggested here by me at the beginning, you can review it, there is an open pull request. I need a little help with a query SQL so maybe it will be possible to make it more precise than it is now (like using gruop by month and then do average for each month). If you or someone wants to help me with the development, you are welcome 🙂

https://github.com/immich-app/immich/pull/6037