authentik timeout error

[tired shore]

Hi,
When I try to link my Immich account to authentik I get

ERROR [Api:ErrorInterceptor~dq5tp3qw] Unknown error: RPError: outgoing request timed out after 30000ms
RPError: outgoing request timed out after 30000ms
at /app/immich/server/node_modules/openid-client/lib/helpers/request.js:140:13
at async Client.requestResource (/app/immich/server/node_modules/openid-client/lib/client.js:1192:22)
at async Client.userinfo (/app/immich/server/node_modules/openid-client/lib/client.js:1289:22)
at async OAuthRepository.getProfile (/app/immich/server/dist/repositories/oauth.repository.js:46:20)
at async AuthService.link (/app/immich/server/dist/services/auth.service.js:181:34)

This only happens when going through external URLs so is suspect NPM to be the issue but I can't figure out which one

I'm using unraid a AIO image

[rancid canopy]

:wave: Hey @tired shore,

Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich .

References

• Container Logs: docker compose logs docs
• Container Status: docker ps -a docs
• Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
• Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA

Checklist

I have...

1. :blue_square: verified I'm on the latest release(note that mobile app releases may take some time).
2. :blue_square: read applicable release notes.
3. :blue_square: reviewed the FAQs for known issues.
4. :blue_square: reviewed Github for known issues.
5. :blue_square: tried accessing Immich via local ip (without a custom reverse proxy).
6. :blue_square: uploaded the relevant information (see below).
7. :blue_square: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable

(an item can be marked as "complete" by reacting with the appropriate number)

Information

In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:

• Your docker-compose.yml and .env files.
• Logs from all the containers and their status (see above).
• All the troubleshooting steps you've tried so far.
• Any recent changes you've made to Immich or your system.
• Details about your system (both software/OS and hardware).
• Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
• The version of the Immich server, mobile app, and other relevant pieces.
• Any other information that you think might be relevant.

Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)

If this ticket can be closed you can use the /close command, and re-open it later if needed.

[vale cedar]

Authelia is publicly accessible via domain name?

[tired shore]

I use authentik and yes, it is exposed

[vale cedar]

Oh sorry. Misread.
Can the immich server curl the authentik domain name?

[tired shore]

Yes it can I tried from container terminal

It's redirecting to authentik and then it just timeout

When trying to link

could it be ipv6?

[vale cedar]

Not sure. I don't use npm for my proxy, someone with more experience may be able to answer.
Can also try and post your configs to make sure all looks good there.

[tired shore]

Could it be because of self signed certificate?

[vale cedar]

Definitely possible, I know the app struggles with them in various areas

[tired shore]

is there a way to disable the verrification ?

[vale cedar]

Not that I know of.

I'm not saying it is the issue. Just could be.

[runic totem]

This is a serverside error, not in the app

What happens if you try in a browser?

[tired shore]

I'm in a browser

Not trying from the IOS App

I get an error after the redirection to authentik when trying to link account

[runic totem]

So what are you seeing in the browser? Like the bot's message mentioned, you need to give us all the relevant information so we can help you

[tired shore]

RPError: outgoing request timed out after 30000ms
at /app/immich/server/node_modules/openid-client/lib/helpers/request.js:140:13
at async Client.requestResource (/app/immich/server/node_modules/openid-client/lib/client.js:1192:22)
at async Client.userinfo (/app/immich/server/node_modules/openid-client/lib/client.js:1289:22)
at async OAuthRepository.getProfile (/app/immich/server/dist/repositories/oauth.repository.js:46:20)
at async AuthService.link (/app/immich/server/dist/services/auth.service.js:181:34)

This is the error i get from the log

AFter clicking link to account it loads and the times out

[runic totem]

So the Immich server isn't able to reach authelia. What's the address you put in?

[tired shore]

But it only happens when using external urls not internal

https://sso.strok.fr/application/o/immich/.well-known/openid-configuration

I tried curl from container terminal and it reaches it

[runic totem]

From inside the immich-server container?

[tired shore]

Yeah the container consol

[runic totem]

That's strange

[tired shore]

curl https://sso.strok.fr/application/o/immich/.well-known/openid-configuration
{
"issuer": "https://sso.strok.fr/application/o/immich/",
"authorization_endpoint": "https://sso.strok.fr/application/o/authorize/",
"token_endpoint": "https://sso.strok.fr/application/o/token/",
"userinfo_endpoint": "https://sso.strok.fr/application/o/userinfo/",
"end_session_endpoint": "https://sso.strok.fr/application/o/immich/end-session/",
"introspection_endpoint": "https://sso.strok.fr/application/o/introspect/",
"revocation_endpoint": "https://sso.strok.fr/application/o/revoke/",
"device_authorization_endpoint": "https://sso.strok.fr/application/o/device/",
"response_types_supported": [
"code",
"id_token",
"id_token token",
"code token",
"code id_token",
"code id_token token"
],
"response_modes_supported": [
"query",
"fragment",
"form_post"
],
"jwks_uri": "https://sso.strok.fr/application/o/immich/jwks/",
"grant_types_supported": [
"authorization_code",
"refresh_token",
"implicit",
"client_credentials",
"password",
"urn:ietf:params:oauth:grant-type:device_code"
],
"id_token_signing_alg_values_supported": [
"RS256"
],
"subject_types_supported": [
"public"
],
"token_endpoint_auth_methods_supported": [
"client_secret_post",
"client_secret_basic"
],
"acr_values_supported": [
"goauthentik.io/providers/oauth2/default"
],
"scopes_supported": [
"openid",
"email",
"profile"
],
"request_parameter_supported": false,
"claims_supported": [
"sub",
"iss",
"aud",
"exp",
"iat",
"auth_time",
"acr",
"amr",
"nonce",
"email",
"email_verified",
"name",
"given_name",
"preferred_username",
"nickname",
"groups"
],
"claims_parameter_supported": false,
"code_challenge_methods_supported": [
"plain",
"S256"
]

This is from container console

[runic totem]

And how are you accessing this console?

[tired shore]

docker exec

It's on unraid so i just have to click the container and the console

[runic totem]

gotcha

Are you absolutely sure the right address is saved in the Immich config?

[tired shore]

Could it be because i have ipv4 and ipv6 records?

[tired shore]

Yeah

[runic totem]

Afaik shouldn't be, but I guess it's maybe possible? idk

[tired shore]

NPM could be the issue

I think$

But what

[tired shore]

Here is the full config

If I change the issuer URL to the local IP of authentik it does work

And It's an outgoing request timout from immich

Is there a way to increase the timeout ?

[runic totem]

That won't make any difference

It's simply not managing to connect to it

Is the system from which you're connecting in the same network?

[tired shore]

Yes

[runic totem]

Can you post the complete error log, not just the stacktrace bit?

ideally with log level on verbose

[tired shore]

Yes

One sec

This is another try (nothing different)

[pine kestrel]

Can you post the compose and env please
Also the output of the curl commands you tried inside the server container , using curl -v

[tired shore]

I'm using the AIO on unraid so no compose file, i'll post the container config

[pine kestrel]

Do you have any local DNS setup or are these all public A records?

[tired shore]

No local DNS (pihole or adguard) and those are all A or AAAA records

[runic totem]

If you do a straight login from the Immich login screen, what happens?

[tired shore]

You mean with OAuth? It fails

This is curl

on the curl if i just add one / at the end it does nothing

Unraid Config

[runic totem]

But at what point/what do the logs say?

Is Immich sending you to authentik at all?

[tired shore]

I'll try again and post the logs

[tired shore]

Yes

[pine kestrel]

curl -v please

[runic totem]

Then the Immich server -is- in fact reaching authentik successfully

(as we can also see with curl)

but that one call is failing for some strange reason

[tired shore]

It's very slow to the first redirection to authentik idk why

[runic totem]

Check your reverse proxy & authentik for logs when there's a timeout. Maybe those have clues

[tired shore]

There is no timeout in authentik

And no error in the reverse proxy

In authentik logs i see the application authorized

[vital nymph]

I haven't read the entire thread but saw you're using unraid

Did you update to 6.12.14?

[tired shore]

Yes

[vital nymph]

Yeaaaaaah

About that

https://forums.unraid.net/topic/181016-61213-to-61214-update-eth0-no-longer-working/

They broke interfaces quite badly in some scenarios

[tired shore]

AHHH

[vital nymph]

Currently also in that position and not 100% how to fix it yet 😅

[tired shore]

I'll try to go back to previous version

[vital nymph]

Didn't help for me (neither did restoring a backup) but it's definitely worth a shot

[tired shore]

So you have the same error with authentik?

[vital nymph]

Not with authentik, maybe it's not even the issue you're having

I'm having issues with HTTPS requests/packets being split up in half for some reason

Not 100% yet 😅

[tired shore]

I'm gonna try to downgrade to see if that solves my issue

[vital nymph]

Definitely a good idea

[tired shore]

As NPM is running as host maybe that's why

[vital nymph]

🤷

[tired shore]

Nope did not solve

[pine kestrel]

Daniel said it didn't solve it for him either tho :/

so it doesn't eliminate it as an issue

NPM shouldn't need to be with host networking tbh

could try switching that

[vital nymph]

Just want to emphasize that I haven't even read the full thread 😅

Was just saying that I ran into network interface issues

That mirrored to docker networks

[pine kestrel]

is this the first time setting up authentik or was it working before?

[tired shore]

For IPV6 to work on unraid apparently it has to be host

[tired shore]

First time

[runic totem]

What you mention about packets getting shredded definitely sounds like it could be this issue though :P

[tired shore]

I'm trying a different reverse proxy

[vital nymph]

In my case it's TLS running EOF

[tired shore]

Now with a differeren NPM container immich can't reach authentik lol

[tired shore]

Well I'm giving up

[verbal oracle]

there are still quite some quirks in docker networks with IPv6, i would try with IPv4 first to verify that works

[tired shore]

I tried but I will try again

My whole network is dual stack and as it works on LAN (which is IPV6+IPV4 as well) I doubt it will fix

[verbal oracle]

try with separate networks with ipv4 only to see if there is a problem

[tired shore]

I'm disabling IPV6 right now on my entire network, server and DNS records

Could it be MTU?

I'll set the MTU back to 1500 as well

Just to try

[tired shore]

Does work

Damn

Might be MTU

Idk

Ill set IPV6 back first and then MTU to see which one block it

[verbal oracle]

K

I personally love ipv6 and learned a lot about it, unfortunately I hear there is issues with it on docker networks. Sometimes it even „bleeds“ into other networks so i disabled ipv6 but not because ipv6 is not cool

MTU can be an issue with certificates often though

So try it and report

[tired shore]

My network used to be IPV4 only and recently switched to dual stack for "future proofing"

MTU has been set to 9000 for a while

IPV6 works

[pine kestrel]

I think v6 in the home lab has limited utility, IDK

v6 is basically good for ISPs and those behind CGNAT and even then with the rate they are handing out blocks I am not confident we won’t have the same issue in a few years

[verbal oracle]

I don’t recommend jumbo frames, many switches don’t support it and fragment, which can be an issue on encrypted traffic or while transferring certs

[tired shore]

I think it's because my router (Provided by ISP so can't change that) is set to 1500

I'll try right now setting back the MTU to 9000

If it fails then MTU it is

[verbal oracle]

In home networks, jumbo frames don’t really give you that much of an advantage

[tired shore]

A bit faster on file transfer over 10G

That's it

Not a lot

[verbal oracle]

If you use them, they need to be consistent everywhere

Yes, on 10gig, a bit… but you will never actually notice the difference without measuring it as you will rarely use 10gig at full speed

[tired shore]

Wellllll video editing from my nas I do notice a light difference

Might be me

[verbal oracle]

Could be if you store it on nvme drives

Keep in mind, secure connections often don’t enjoy being fragmented:P

[tired shore]

I have a ZFS RAIDz1 array of 4 8TB nvme

It's MTU

Set it back and it fails

Ciao Jumbo Frames

[verbal oracle]

Probably some certificate verification that gets fragmented

[tired shore]

Idk but for now I'll just disable Jumbo frames$

I'll ask my ISP if they can set the router MTU to 9000

[verbal oracle]

I still wouldn’t, maybe in a storage backend like only… or Therese you need to make sure every device on the network does jumbo frames

Or you might run into issues eventually, 95% of which can be fixed by manipulating the mss in tcp handshakes

The rest you will just not know what the issue will is, lol

I’ve had my fair share of troubles with MTU

[tired shore]

At least it is now fixed

Thaks a lot to everyone