Latest update causing reboot loop | Immich | Page 1

pure vigil Sep 19, 2024, 6:43 PM

#

Hi all- thanks in advance for the help. I am running under Docker windows (sorry lol). I've been running immich for the last couple months without problems. However, after the latest update, the immich_server component is stuck in a reboot loop. It repeatedly exits, gets restarted, and then exits again.

Logs, .env, and YML attached. Appreciate the help

📎 env 📎 docker-compose.yml 📎 log.log

torn stormBOT Sep 19, 2024, 6:43 PM

#

:wave: Hey @pure vigil,

Thanks for reaching out to us. Please follow the recommended actions below; this will help us be more effective in our support effort and leave more time for building Immich immich .

References

Container Logs: docker compose logs docs
Container Status: docker compose ps docs
Reverse Proxy: https://immich.app/docs/administration/reverse-proxy

Checklist

:ballot_box_with_check: I have verified I'm on the latest release(note that mobile app releases may take some time).
:ballot_box_with_check: I have read applicable release notes.
:ballot_box_with_check: I have reviewed the FAQs for known issues.
:ballot_box_with_check: I have reviewed Github for known issues.
:ballot_box_with_check: I have tried accessing Immich via local ip (without a custom reverse proxy).
:ballot_box_with_check: I have uploaded the relevant logs, docker compose, and .env files, making sure to use code formatting.
:ballot_box_with_check: I have tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable

(an item can be marked as "complete" by reacting with the appropriate number)

If this ticket can be closed you can use the /close command, and re-open it later if needed.

torn stormBOT Sep 19, 2024, 6:45 PM

#

torn storm :wave: Hey <@271466848451690497>, Thanks for reaching out to us. Please follow ...

Successfully submitted, a tag has been added to inform contributors. :white_check_mark:

mighty needle Sep 19, 2024, 6:46 PM

#

Alright first things first, update your redis container, it's still d6c2911ac51b289db208767581a5d154544f2b2fe4914ea5056443f62dc6e900 but newest is 2d1463258f2764328496376f5d965f20c6a67f66ea2b06dc42af351f75248792

#

It's a little annoying, but check the compose file in each release for an update 🙂

pure vigil Sep 19, 2024, 6:50 PM

#

mighty needle It's a little annoying, but check the compose file in each release for an update...

this is going to sound stupid, but I am not super with-it when it comes to docker. How do I update it? is docker compose pull not performing the update?

mighty needle Sep 19, 2024, 6:51 PM

#

The compose file contains a line with the redis container:
image: docker.io/redis:6.2-alpine@sha256:d6c2911ac51b289db208767581a5d154544f2b2fe4914ea5056443f62dc6e900
You need to replace that with
image: docker.io/redis:6.2-alpine@sha256:2d1463258f2764328496376f5d965f20c6a67f66ea2b06dc42af351f75248792

#

After that, just do docker compose up -d again

#

And no, the redis container isn't tagged with release or such I'm afraid, it's from another project

pure vigil Sep 19, 2024, 6:53 PM

#

ok - that is done, but still same problem.

#

it did pull a new redis, but immich_server still exits

mighty needle Sep 19, 2024, 6:54 PM

#

Try a simple docker restart immich_server

#

If that doesn't work let's increase verbosity by putting IMMICH_LOG_LEVEL=verbose in your .env file, and then compose down/compose up

pure vigil Sep 19, 2024, 6:57 PM

#

OK - I got the actal error, but it makes no sense because nothing has changed on my system or in my config from before the update.

#

immich_server | [Nest] 19 - 09/19/2024, 11:56:32 AM ERROR [Api:StorageService] Failed to write upload/encoded-video/.immich: Error: EACCES: permission denied, open 'upload/encoded-video/.immich'
immich_server | [Nest] 19 - 09/19/2024, 11:56:32 AM ERROR [Api:StorageService] The "encoded-video" folder cannot be written to, please make sure the volume is mounted with the correct permissions
immich_server | api worker exited with code 1

rocky viper Sep 19, 2024, 6:57 PM

#

We just added a check for this in the latest release

#

Like the error says, it indicates a permission issue with your UPLOAD_LOCATION

mighty needle Sep 19, 2024, 6:58 PM

#

The latest release wants to make an .immich file at the top level

#

Likely your permissions for that very directory aren't fully compliant with that

pure vigil Sep 19, 2024, 6:58 PM

#

ok let me just 777 it and see what happens. hold on a sec.

mighty needle Sep 19, 2024, 6:58 PM

#

lol, the old "777 should fix it"

pure vigil Sep 19, 2024, 7:03 PM

#

sooo... these are the permissions on that directory, already 😕

drwxrwxrwx+

#

and oddly there is already a .immich file there

#

-rwxrwxrwx+ 1 home wheel 13B Sep 17 11:03 .immich

mighty needle Sep 19, 2024, 7:08 PM

#

Is this maybe somehow a CIFS UID GID thing

pure vigil Sep 19, 2024, 7:09 PM

#

given it's 777 though, immich should be able to write to it. And, apparently, it already did the first time I pulled the update, right? because the timestamp on that file is Sept-17

#

should i remove .immich and see if it recreates it?

mighty needle Sep 19, 2024, 7:10 PM

#

What if you delete the file, does it get recreated? :

#

Lmao

pure vigil Sep 19, 2024, 7:10 PM

#

^ lol lemme see

mighty needle Sep 19, 2024, 7:10 PM

#

There's one in every folder just fyi

pure vigil Sep 19, 2024, 7:11 PM

#

ok - same exact error. claims it can't be written to. but it DID recreate the file.
13B Sep 19 12:10 .immich

mighty needle Sep 19, 2024, 7:16 PM

#

Well I'm out of ideas for now

#

Still going for CIFS shenanigans

pure vigil Sep 19, 2024, 7:18 PM

#

if CIFS is broken I basically can't use immich which is a bummer. Everything is saved on an SMB share with a password and this was the only way I was able to get the share mounted (granted I'm using docker windows but my file server is FreeBSD/FreeNAS).

Is there any way to circumvent the check, since obviously it can write there (and has been)

mighty needle Sep 19, 2024, 7:18 PM

#

I'm not saying it's broken I'm saying you might need more options

pure vigil Sep 19, 2024, 7:19 PM

#

open to suggestions if you (or anyone else) has them 🤞

mighty needle Sep 19, 2024, 7:19 PM

#

I see people adding crazy things like ,vers=3.0,file_mode=0777,dir_mode=0777

pure vigil Sep 19, 2024, 7:20 PM

#

will try rn

#

no luck

mighty needle Sep 19, 2024, 7:23 PM

#

I'm out of ideas for you and I don't want to fudge it all up

pure vigil Sep 19, 2024, 7:24 PM

#

https://github.com/immich-app/immich/issues/12630

ebon fulcrum Sep 19, 2024, 7:24 PM

#

Did you set UPLOAD_LOCATION itself to 777, or just its child folders?

pure vigil Sep 19, 2024, 7:25 PM

#

it's also 777

#

drwxrwxrwx+ 7 user wheel 7B Jul 5 20:55 upload

ebon fulcrum Sep 19, 2024, 7:30 PM

#

Are you still getting the same error?

#

Or is it a different error?

pure vigil Sep 19, 2024, 7:31 PM

#

same error

#

people on that thread are saying to touch/create the .immich file, but my issue is a little worse, bc immich actually can create the file but acts like it can't.

Guy who created profile/.immich says after he did that though problem just moved to another directory. <shrug>

ebon fulcrum Sep 19, 2024, 7:34 PM

#

Can you touch that file manually and see if that changes anything?

pure vigil Sep 19, 2024, 7:34 PM

#

sure lemme try

#

0B Sep 19 12:34 immich

#

lemme restart.

#

same error. lol. it overwrote it and added data

#

13B Sep 19 12:35 .immich

#

and then said it couldn't write

ebon fulcrum Sep 19, 2024, 7:37 PM

#

……what is happening

pure vigil Sep 19, 2024, 7:38 PM

#

tldr CIFS is evil. but this sucks honestly 😕

wise spear Sep 19, 2024, 7:39 PM

#

I had issues with it as well. I had to manually go in and create each .immich file (SMB Share), then worked fine after without changing anything else

pure vigil Sep 19, 2024, 7:39 PM

#

wise spear I had issues with it as well. I had to manually go in and create each .immich fi...

did you do this in a different way? I'm on the machine hosting the share and touching the file. No fix. Are you doing it differently?

wise spear Sep 19, 2024, 7:40 PM

#

The only difference I think I was I did it from the machine connecting to the share

ebon fulcrum Sep 19, 2024, 7:41 PM

#

We tried to reproduce this and couldn’t and figured it was a case of the UPLOAD_LOCATION folder not having the right permissions. But this is clearly different

pure vigil Sep 19, 2024, 7:41 PM

#

wise spear The only difference I think I was I did it from the machine connecting to the sh...

just tried this in notepad by saving an empty file called .immich -- no dice.

ebon fulcrum Sep 19, 2024, 7:41 PM

#

Is the cifs share on windows?

pure vigil Sep 19, 2024, 7:42 PM

#

the share is running on a FreeNAS machine (freebsd). Windows (running docker) connects with SMB

ebon fulcrum Sep 19, 2024, 7:43 PM

#

Try ssh’ing into the freenas machine and touching the file from there

pure vigil Sep 19, 2024, 7:43 PM

#

yah that's what i've been doing

#

just was humoring @wise spear by trying notepad

ebon fulcrum Sep 19, 2024, 7:43 PM

#

Ah, gotcha

#

My only lead is that smb has some special handling with dot files to treat them as hidden, and on windows opening a hidden file with w mode is an error. I don’t know if this is the issue here but I can’t think of anything else

#

I’ve personally made a test instance in wsl2 connecting to a mounted truenas scale smb share and it had no issues. I wonder if it’s a difference in the protocol version or maybe a different share configuration

#

There’s also the share ACL besides the unix file permissions. Maybe it’s something to do with that?

pure vigil Sep 19, 2024, 8:00 PM

#

yah idk. haven't issues like this before in other contexts. usually if you can create a file you're good

ebon fulcrum Sep 19, 2024, 8:17 PM

#

Besides the share ACL, I’ve got nothing

#

What a weird issue

pure vigil Sep 19, 2024, 8:17 PM

#

yup. i will keep an eye on https://github.com/immich-app/immich/discussions/12798

ebon fulcrum Sep 19, 2024, 8:55 PM

#

Can you run docker compose down to stop the containers, docker compose run --entrypoint=bash immich-server to get a shell with the server container and check upload/encoded-video/.immich?

pure vigil Sep 19, 2024, 9:11 PM

#

ebon fulcrum Can you run `docker compose down` to stop the containers, `docker compose run -...

Will try this in a bit

cloud lantern Sep 19, 2024, 9:16 PM

#

So to be clear, you're running FreeNAS (TrueNAS Core?) as the OS your share exists in. Would you be able share the config for that share, and the mount config that you have?

#

I want to try and get a reproduction of this, but any attempts at that so far have led nowhere

pure vigil Sep 19, 2024, 9:30 PM

#

ebon fulcrum Can you run `docker compose down` to stop the containers, `docker compose run -...

📎 console.txt

pure vigil Sep 19, 2024, 9:35 PM

#

cloud lantern So to be clear, you're running FreeNAS (TrueNAS Core?) as the OS your share exis...

TrueNAS Core 13.0-U6.1; immich is being run on windows under docker desktop (because reasons, but at this point really wish I wasn't).

this is how the CIFS mount is configured in the YML:

volumes:
nas-share:
driver_opts:
type: cifs
o: "username=username,password=password,addr=192.168.25.240,vers=3.0,file_mode=0777,dir_mode=0777"
device: "//SERVERNAME/ShareName/immich/upload"
model-cache:

pure vigil Sep 19, 2024, 9:35 PM

#

pure vigil TrueNAS Core 13.0-U6.1; immich is being run on windows under docker desktop (bec...

2024-09-19_14_32_13-TrueNAS_-_freenas.local_Mozilla_Firefox.png

cloud lantern Sep 19, 2024, 9:35 PM

#

TIL (or remembered) you can do CIFS directly in compose

#

Thanks for that, I will test your setup next if I can't get a reproduction working on my own setup 😛

cloud lantern Sep 19, 2024, 10:51 PM

#

Recreated locally with your exact mount settings (using unraid for the NAS) and everything works fine here

#

Could you please provide an ls -la output from the NAS itself?

pure vigil Sep 19, 2024, 10:53 PM

#

https://github.com/immich-app/immich/discussions/12798#discussioncomment-10696358

#

And this is the output inside encoded-video — in fact this is the .immich file that .immich created and says it can’t write 😵‍💫

-rwxrwxrwx+ 1 user wheel 13B Sep 19 14:41 .immich

I’m just really at a loss

#

Maybe this is helpful info, too;

#1286397327875244043 message

cloud lantern Sep 19, 2024, 10:59 PM

#

I guess it's time for me to spin up truenas core...

clear wadi Sep 19, 2024, 11:05 PM

#

pure vigil And this is the output inside encoded-video — in fact this is the .immich file t...

This is on the host itself not the share?

pure vigil Sep 19, 2024, 11:07 PM

#

clear wadi This is on the host itself not the share?

What you’re quoting is the output from FreeNAS which is hosting the share.

#

This (below) is the output from a bash console inside immich-server

#1286397327875244043 message

clear wadi Sep 19, 2024, 11:09 PM

#

pure vigil What you’re quoting is the output from FreeNAS which is hosting the share.

So it is the output from the underlying filesystem and not the share itself, right?

clear wadi Sep 19, 2024, 11:10 PM

#

pure vigil This (below) is the output from a bash console inside immich-server https://di...

This looks like it is working fine.

pure vigil Sep 19, 2024, 11:11 PM

#

clear wadi This looks like it is working fine.

Right. I agree. Immich should start up. But when it’s running its checks it thinks it can’t write. Even though it does, in fact, write.

#

And I’ve been running it for months and it’s been creating files and folders like there’s no tomorrow

clear wadi Sep 19, 2024, 11:12 PM

#

pure vigil Right. I agree. Immich should start up. But when it’s running its checks it thin...

I think you need to actually open the file and edit it. Not just touch it.

cloud lantern Sep 19, 2024, 11:12 PM

#

pure vigil Right. I agree. Immich should start up. But when it’s running its checks it thin...

can you show me the user permissions for the user you're connecting with for the cifs share?

clear wadi Sep 19, 2024, 11:12 PM

#

I'm guessing this is related to some stupid quirk with hidden files.

cloud lantern Sep 19, 2024, 11:12 PM

#

Trying to replicate your setup as close as possible

pure vigil Sep 19, 2024, 11:13 PM

#

clear wadi I think you need to actually open the file and edit it. Not just touch it.

I can try this. What should the content be? Anything? When Immich writes to .immich I swear the file size change

pure vigil Sep 19, 2024, 11:13 PM

#

cloud lantern can you show me the user permissions for the user you're connecting with for the...

Yah give me a second

clear wadi Sep 19, 2024, 11:14 PM

#

pure vigil I can try this. What should the content be? Anything? When Immich writes to .imm...

It's just a number

pure vigil Sep 19, 2024, 11:19 PM

#

cloud lantern Trying to replicate your setup as close as possible

This is what it currently looks like. I have it set to allow everyone but to specifically deny user2. I am going to check, really quick, if I get rid of the deny for user2 if anything changes, but it shouldn't. (edit: no change)

cloud lantern Sep 19, 2024, 11:21 PM

#

Right, that's fine, what about the user settings specifically for the user you're connecting from?

#

This screen

pure vigil Sep 19, 2024, 11:22 PM

#

#

same thing bigger.

cloud lantern Sep 19, 2024, 11:25 PM

#

Where do the ACL configs live lmao

#

Ah, found it immediately after sending that...

#

Ah, no this is just the share ACL which looks different to the view you had

pure vigil Sep 19, 2024, 11:26 PM

#

storage pool ACL

#

sharing>smb>three dots>file system acl

cloud lantern Sep 19, 2024, 11:30 PM

#

Nice,t y

pure vigil Sep 19, 2024, 11:31 PM

#

clear wadi It's just a number

this did not work

ebon fulcrum Sep 19, 2024, 11:32 PM

#

So you can make the file, but can't write into it?

clear wadi Sep 19, 2024, 11:33 PM

#

It must be that stupid Windows thing with opening a hidden file with w flag

#

Can you open and write to immich (no leading period)?

cloud lantern Sep 19, 2024, 11:34 PM

#

So with your exact setup on truenas core, everything works as expected 😅

pure vigil Sep 19, 2024, 11:34 PM

#

clear wadi Can you open and write to `immich` (no leading period)?

h/o i think miscomunication. immich didn't start says it cant write the .immich file that i created with content

cloud lantern Sep 19, 2024, 11:34 PM

#

Not using wsl though, but mert did test that earlier

pure vigil Sep 19, 2024, 11:34 PM

#

cloud lantern So with your exact setup on truenas core, everything works as expected 😅

bruh.

#

what was causing the issue for the other people? I saw they solved it by creating the file manually? is this related possibly?

#

idk why you can't recreate

#

maybe wsl?

cloud lantern Sep 19, 2024, 11:37 PM

#

Realistically when you're connecting from WSL -> TrueNAS core, the fact it's WSL and running on windows shouldn't even be a consideration

pure vigil Sep 19, 2024, 11:37 PM

#

this might be the thing that causes me to wipe the entire computer and install a linux image but ffs...

wise spear Sep 19, 2024, 11:37 PM

#

Mine was it saying it couldn't create the file. So I created manually and then it started fine. Didn't touch any permission stuff, still could upload photos with no problems. I'm using truenas scale as the smb host and running docker on Linux

cloud lantern Sep 19, 2024, 11:37 PM

#

As the docker engine itself is running within WSL

pure vigil Sep 19, 2024, 11:37 PM

#

right

clear wadi Sep 19, 2024, 11:38 PM

#

But smb can have windows compat settings for hidden files is what I meant

#

Not that Windows is actually in play.

pure vigil Sep 19, 2024, 11:38 PM

#

yah is this a wsl incompatibility because it's called (dot)immich

#

instead of like writetest.txt

#

i hate hate hate docker windows

cloud lantern Sep 19, 2024, 11:39 PM

#

#

To confirm everything created successfully

clear wadi Sep 19, 2024, 11:39 PM

#

pure vigil instead of like writetest.txt

You could write this file successfully from inside the container?

pure vigil Sep 19, 2024, 11:39 PM

#

clear wadi You could write this file successfully from inside the container?

i can write .immich within the container too. h/o let me do it again and screenshot

clear wadi Sep 19, 2024, 11:40 PM

#

Can you write a new dot file as well? Like .immich2?

clear wadi Sep 19, 2024, 11:41 PM

#

pure vigil this did not work

I thought you said it didn't work though

pure vigil Sep 19, 2024, 11:41 PM

#

clear wadi I thought you said it didn't work though

meant immich didn't start. miscommunication

#

cloud lantern Sep 19, 2024, 11:42 PM

#

You're root

clear wadi Sep 19, 2024, 11:42 PM

#

So reading and writing everything inside the container all works

cloud lantern Sep 19, 2024, 11:42 PM

#

su node

pure vigil Sep 19, 2024, 11:42 PM

#

cloud lantern `su node`

you solved.

clear wadi Sep 19, 2024, 11:42 PM

#

We run as root by default, is OP running as 1000?

pure vigil Sep 19, 2024, 11:42 PM

#

cloud lantern Sep 19, 2024, 11:43 PM

#

clear wadi We run as root by default, is OP running as 1000?

The server runs as node by default does it not?

clear wadi Sep 19, 2024, 11:43 PM

#

cloud lantern The server runs as node by default does it not?

Nope

#

I don't think so

cloud lantern Sep 19, 2024, 11:43 PM

#

Hmmm, yea it is running as root by default

#

https://tenor.com/bWSfU.gif

Tenor

pure vigil Sep 19, 2024, 11:44 PM

#

so the server has been writing for months to this folder. maybe the write test is different

#

i can't write as node. anything

clear wadi Sep 19, 2024, 11:47 PM

#

That's expected. You aren't running this with a specific id are you? It will default to root

#

If it was as node it would never be able to create anything like all uploads would fail

cloud lantern Sep 19, 2024, 11:47 PM

#

Alright lets try something

#

Go back to root, run the following

#

node

pure vigil Sep 19, 2024, 11:48 PM

#

clear wadi That's expected. You aren't running this with a specific id are you? It will def...

no specific ID afaik. immich was basically my only need for docker, and I didn't have a linux machine available, so i just installed docker windows, accepted the defaults, pulled, and was pretty much up and running

cloud lantern Sep 19, 2024, 11:48 PM

#

const { default: fs } = await import("node:fs/promises")

#

await fs.writeFile('/usr/src/app/upload/encoded-video/.immich1', 'test')

clear wadi Sep 19, 2024, 11:48 PM

#

Is it failing on the encoded video check still when it tries to boot?

pure vigil Sep 19, 2024, 11:48 PM

#

const results in undefined

cloud lantern Sep 19, 2024, 11:48 PM

#

pure vigil const results in undefined

Yea that's fine

pure vigil Sep 19, 2024, 11:49 PM

#

await also, undefined

clear wadi Sep 19, 2024, 11:49 PM

#

Type node first lol

cloud lantern Sep 19, 2024, 11:49 PM

#

clear wadi Type `node` first lol

It'll still show undefined

#

There's nothing returned from those 😛

pure vigil Sep 19, 2024, 11:49 PM

#

cloud lantern Sep 19, 2024, 11:49 PM

#

Check the encoded-video folder pls

#

Does a .immich1 file exist with contents 'test'?

pure vigil Sep 19, 2024, 11:50 PM

#

content didn't change

#

nvm

#

my bad

#

YES

cloud lantern Sep 19, 2024, 11:50 PM

#

So it wrote that?

pure vigil Sep 19, 2024, 11:50 PM

#

yes

cloud lantern Sep 19, 2024, 11:50 PM

#

Well that's just extra confusing lmao

#

Wtaf

clear wadi Sep 19, 2024, 11:50 PM

#

Can you try to start it again?

pure vigil Sep 19, 2024, 11:51 PM

#

yes

#

#

#

it wrote 1726789955046 to .immich

clear wadi Sep 19, 2024, 11:53 PM

#

So it successfully writes and then throws an error?

pure vigil Sep 19, 2024, 11:53 PM

#

yup

cloud lantern Sep 19, 2024, 11:53 PM

#

This is honestly baffling

#

Because it isn't even failing at the read, it's failing on the write, while succeeding to write the file...

pure vigil Sep 19, 2024, 11:54 PM

#

idk how you guys coded the error. it says filed write, but then says error: eacces: permission denied, open

#

maybe it is wsl with hidden file? idk though

cloud lantern Sep 19, 2024, 11:54 PM

#

The last part is just raw what node returns to us

pure vigil Sep 19, 2024, 11:54 PM

#

ah

ebon fulcrum Sep 19, 2024, 11:56 PM

#

Maybe the first write works, but the second write fails?

cloud lantern Sep 19, 2024, 11:56 PM

#

There is only one write

#

Oh, you mean on retry

#

Sure, we can test that

#

Can you open up
node

#

const { default: fs } = await import("node:fs/promises")

#

await fs.writeFile('/usr/src/app/upload/encoded-video/.immich1', 'test1234')

ebon fulcrum Sep 19, 2024, 11:56 PM

#

It calls verifyWriteAccess twice the first time it’s writing the file

pure vigil Sep 19, 2024, 11:58 PM

#

#

cloud lantern Sep 19, 2024, 11:58 PM

#

Had you deleted it since last time?

pure vigil Sep 19, 2024, 11:58 PM

#

yup

cloud lantern Sep 19, 2024, 11:58 PM

#

Just because the .immich file is gone now

pure vigil Sep 19, 2024, 11:59 PM

#

i emptied everything to test

cloud lantern Sep 19, 2024, 11:59 PM

#

Oh, we need to test overwriting it

#

So run the commands again, change the value to 12345

pure vigil Sep 19, 2024, 11:59 PM

#

ok

#

yup

#

cloud lantern Sep 20, 2024, 12:00 AM

#

Excellent, we have a reproduction!

pure vigil Sep 20, 2024, 12:00 AM

#

oh wait

#

hoooooold on

#

i have notepad open

#

fml

cloud lantern Sep 20, 2024, 12:00 AM

#

Hmmm, that's interesting actually

ebon fulcrum Sep 20, 2024, 12:00 AM

#

But that might be relevant though. Maybe there’s some kind of file lock from the earlier fs calls

cloud lantern Sep 20, 2024, 12:00 AM

#

Could we be writing to quickly in succession?

#

Yea.....

#

SMB loves to lock files lol

pure vigil Sep 20, 2024, 12:01 AM

#

ok still happening. wasn't my fault

#

cloud lantern Sep 20, 2024, 12:01 AM

#

Ok, could you try writing to immich1 instead twice in a row?

#

No dot

pure vigil Sep 20, 2024, 12:01 AM

#

yah

#

give me a sec

cloud lantern Sep 20, 2024, 12:01 AM

#

undefined is a success fyi

pure vigil Sep 20, 2024, 12:02 AM

#

first attempt worked:

#

cloud lantern Sep 20, 2024, 12:02 AM

#

And second?

pure vigil Sep 20, 2024, 12:03 AM

#

yep, worked.

cloud lantern Sep 20, 2024, 12:03 AM

#

Yea, so it's absolutely a dotfile thing

#

Great

#

@clear wadi do we just say fuck this and change it to not a dotfile

#

I can't reproduce this problem, I have no idea why, but it's obviously dotfile related

clear wadi Sep 20, 2024, 12:03 AM

#

Why do we think it is a dot?

cloud lantern Sep 20, 2024, 12:04 AM

#

He just reproduced the error in the node console

#

Writing the second time fails with a dotfile

#

It doesn't with a normal file

#

Same folder

clear wadi Sep 20, 2024, 12:04 AM

#

When it was open in notepad right?

cloud lantern Sep 20, 2024, 12:04 AM

#

No

#

He closed that and tried another 5 times

#

zot do a fresh test with .immich2 twice in a row just to confirm?

pure vigil Sep 20, 2024, 12:05 AM

#

yah i just tried again. repro'd again

cloud lantern Sep 20, 2024, 12:05 AM

#

Cool ty

#

Ahead of me 😉

clear wadi Sep 20, 2024, 12:05 AM

#

Can we try pasting 3 lines? Write, read, write

pure vigil Sep 20, 2024, 12:05 AM

#

clear wadi Sep 20, 2024, 12:05 AM

#

And then do it with a dot and a non dot file and post the screenshot?

cloud lantern Sep 20, 2024, 12:05 AM

#

I mean, this seems pretty clear there's a problem with the dotfiles specifically

#

But we can test a read too

pure vigil Sep 20, 2024, 12:06 AM

#

non dot:

clear wadi Sep 20, 2024, 12:06 AM

#

Can you reproduce Zack?

pure vigil Sep 20, 2024, 12:06 AM

#

i don't know node. how should i read it out

cloud lantern Sep 20, 2024, 12:06 AM

#

(await fs.readFile('/usr/src/app/upload/encoded-video/.immich1')).toString()

cloud lantern Sep 20, 2024, 12:07 AM

#

clear wadi Can you reproduce Zack?

Nope

pure vigil Sep 20, 2024, 12:07 AM

#

cloud lantern Sep 20, 2024, 12:07 AM

#

So we can write, and we can read, but we can't overwrite

clear wadi Sep 20, 2024, 12:07 AM

#

The start up code does write, read, write the first time

clear wadi Sep 20, 2024, 12:07 AM

#

cloud lantern So we can write, and we can read, but we can't overwrite

Is it every overwrite?

pure vigil Sep 20, 2024, 12:07 AM

#

cloud lantern Sep 20, 2024, 12:07 AM

#

Yep

#

Every overwrite

clear wadi Sep 20, 2024, 12:08 AM

#

So basically open with w fails

cloud lantern Sep 20, 2024, 12:08 AM

#

Yea, like the docs mert posted said

clear wadi Sep 20, 2024, 12:08 AM

#

And this works fine without a dot file

cloud lantern Sep 20, 2024, 12:08 AM

#

Yep

clear wadi Sep 20, 2024, 12:09 AM

#

Yeah I think that sounds like the problem.

#

Let's transition to not a dot file

cloud lantern Sep 20, 2024, 12:09 AM

#

New cursed docs entry 🙂

#

dotfiles are cursed (on smb)

pure vigil Sep 20, 2024, 12:09 AM

#

i love you guys if you just solved this...... where's the cookie jar $$

But seriously fuck docker windows

cloud lantern Sep 20, 2024, 12:09 AM

#

I still want to reproduce this...

#

I'm guessing it must be a WSL thing

#

Which is... super weird

clear wadi Sep 20, 2024, 12:10 AM

#

cloud lantern I still want to reproduce this...

It has to be the smb config right?

cloud lantern Sep 20, 2024, 12:10 AM

#

Let me spin up a windows VM

cloud lantern Sep 20, 2024, 12:10 AM

#

clear wadi It has to be the smb config right?

I have a fresh truenas core setup, with the exact same settings as them, and the exact same mount settings in compose

#

And yet it doesn't reproduce on linux

pure vigil Sep 20, 2024, 12:11 AM

#

it's only in node that it's throwing an error; since i can do whatever i want to the file from the command line

#

as root*

cloud lantern Sep 20, 2024, 12:12 AM

#

pure vigil it's only in node that it's throwing an error; since i can do whatever i want to...

Btw it is worth noting you could just use NFS to connect from docker 😛

#

Don't have to use CIFS just because you're using WSL

#

NFS will work from docker over there too

pure vigil Sep 20, 2024, 12:13 AM

#

cloud lantern Btw it is worth noting you could just use NFS to connect from docker 😛

my SMB share has a username/password and for whatever reason i couldn't get it to work using NFS

cloud lantern Sep 20, 2024, 12:13 AM

#

I see

pure vigil Sep 20, 2024, 12:13 AM

#

should've been my first clue wsl was cursed...

cloud lantern Sep 20, 2024, 12:13 AM

#

What windows version are you running?

quartz spindle Sep 20, 2024, 12:14 AM

#

news flash: docker on windows sucksss

#

thanks guys

cloud lantern Sep 20, 2024, 12:14 AM

#

I'm now going to spin up a windows VM lmao to replicate that part of your setup

pure vigil Sep 20, 2024, 12:14 AM

#

Edition Windows 10 Pro
Version 22H2
Installed on ‎6/‎18/‎2024
OS build 19045.4529
Experience Windows Feature Experience Pack 1000.19058.1000.0

cloud lantern Sep 20, 2024, 12:14 AM

#

Hmmm, can I WSL within a VM...

#

I think I have nested virtualisation enabled wherever I need to, we'll see I guess

#

Who doesn't love spinning up windows VMs at 1:15AM 🙂

#

Want to say thanks to you too @pure vigil for being so responsive to allow us to actually reproduce the problem

#

I spent a lot of my day today trying to reproduce this with no success

pure vigil Sep 20, 2024, 12:19 AM

#

dude, no, thank YOU. Immich is saving my life and I am surprised by how responsive you guys are. If this isn't just me being an idiot and if you're able to fix it and I get Immich back... totally worth helping you guys repro

#

fwiw, i just tried this and repro'd again. after the second test, i renamed the file test1 to .test1 and it won't overwrite it. so yah. confirmed, at least FOR ME, that it's because it's a dot file.

Hopefully you can reproduce.

#

@cloud lantern check this out

#

if you change the properties in windows and un-check hidden, you can overwrite it. once you overwrite it, it gets hidden attribute again; can't be re-overwritten

cloud lantern Sep 20, 2024, 12:25 AM

#

Yeaaaaaa

ebon fulcrum Sep 20, 2024, 12:26 AM

#

yeah screw that

cloud lantern Sep 20, 2024, 12:26 AM

#

I still don't really get how WSL is affected by this though

#

Thought WSL is a weird concoction 😛

#

I should have a windows VM in 5 or so minutes

pure vigil Sep 20, 2024, 12:27 AM

#

this is gonna make me build a linux machine. wife gonna kill me another computer.

ebon fulcrum Sep 20, 2024, 12:28 AM

#

Docker can run from powershell etc. too. Maybe as far as samba is concerned, docker is mounting the share as a windows client

cloud lantern Sep 20, 2024, 12:32 AM

#

The files show up as hidden for me too fyi

#

But yea, maybe linux docker doesn't care

cloud lantern Sep 20, 2024, 12:50 AM

#

Right, booting immich in WSL now

#

Only 274 seconds to pull the image

#

And it's still going, just frozen 🙂

pure vigil Sep 20, 2024, 12:54 AM

#

the last 300kib..

cloud lantern Sep 20, 2024, 12:54 AM

#

Lmao, and it worked just fine

pure vigil Sep 20, 2024, 12:54 AM

#

lol seriously

#

what the actual f

#

you're in powershell? and cifs?

cloud lantern Sep 20, 2024, 12:56 AM

#

Powershell?

#

No I ran this directly in WSL

#

You're running the compose commands from powershell?

pure vigil Sep 20, 2024, 12:56 AM

#

no

#

from CMD

#

and i used docker desktop

cloud lantern Sep 20, 2024, 12:57 AM

#

windows command prompt?

pure vigil Sep 20, 2024, 12:57 AM

#

yup

cloud lantern Sep 20, 2024, 12:57 AM

#

Can you verify some settings for me?

pure vigil Sep 20, 2024, 12:57 AM

#

sure

cloud lantern Sep 20, 2024, 12:57 AM

#

If you go into docker desktop

#

settings > resources > WSL integration

#

What does it show?

pure vigil Sep 20, 2024, 12:58 AM

#

cloud lantern Sep 20, 2024, 12:58 AM

#

Advanced?

pure vigil Sep 20, 2024, 12:58 AM

#

cloud lantern Sep 20, 2024, 12:59 AM

#

Can you open powershell and run
wsl --list

pure vigil Sep 20, 2024, 1:00 AM

#

PS C:\Users\NUC8-i3BEH1> wsl --list
Windows Subsystem for Linux Distributions:
docker-desktop-data (Default)
docker-desktop
PS C:\Users\NUC8-i3BEH1>

cloud lantern Sep 20, 2024, 1:04 AM

#

Works through CMD too lol

#

But I did install ubuntu WSL as my default

#

So I think it's running in that

#

Maybe it's different if I hadn't done that?

pure vigil Sep 20, 2024, 1:06 AM

#

cloud lantern But I did install ubuntu WSL as my default

i definitely didn't do this. since this was my only use case for docker i just took whatever defaults docker pushes when you install docker desktop

cloud lantern Sep 20, 2024, 1:06 AM

#

Hmmm

#

Let me see if I can revert to that

#

The performance is actually terrible

#

Crazy to me that so many people run setups like this 🤣

pure vigil Sep 20, 2024, 1:14 AM

#

It’s atrocious and I hate it. I have like two things that won’t natively run under BSD and this is one of them (bc of docker). The easy thing for me is if you can repro this and fix it (maybe by renaming the test file). But if you can’t repro it and I am like the edge case’s edge case, I am gonna have to figure something else out. Docker desktop in windows is hot garage. It’s so painful to use

cloud lantern Sep 20, 2024, 1:15 AM

#

It probably doesn't help that I am double virtualising here

#

But I don't think that's really the problem 😛

#

I still can't reproduce it though lmao

pure vigil Sep 20, 2024, 1:19 AM

#

Ok I mean you have been super generous with your time. I am bewildered bc it’s clear what’s happening on my system re refusing to overwrite hidden files. But if you can’t reproduce it 🤷‍♂️

Honestly bizarre

cloud lantern Sep 20, 2024, 1:19 AM

#

I think we'll still make the change

#

I mostly wanted a reproduction case so we could understand it further potentially 😛

pure vigil Sep 20, 2024, 1:20 AM

#

The zot fix… affected user(s): 1

cloud lantern Sep 20, 2024, 1:29 AM

#

I assume you don't have any special config in the truenas core smb service settings?

#

#

Services > SMB

pure vigil Sep 20, 2024, 1:35 AM

#

SMB 2/3 extensions

cloud lantern Sep 20, 2024, 1:35 AM

#

Enable Apple SMB2/3 Protocol Extensions?

#

You have that turned on?

pure vigil Sep 20, 2024, 1:35 AM

#

Yep

clear wadi Sep 20, 2024, 1:49 AM

#

I think the windows settings are in the smb stuff.

#

Like the behavior we are seeing is via the smb server itself not the client.

cloud lantern Sep 20, 2024, 1:50 AM

#

cloud lantern

This is the server

clear wadi Sep 20, 2024, 1:51 AM

#

What about aux params?

#

Idk, there are the actual underlying settings, not just what true nas shows you

cloud lantern Sep 20, 2024, 1:51 AM

#

TrueNAS applies the defaults

#

Which in this case is "hidden dot files = true"

#

But I just tried applying that manually even, and it still doesn't break for me...

clear wadi Sep 20, 2024, 1:52 AM

#

With same file mask?

#

Can you get all the raw settings and compare them or something like that?

pure vigil Sep 20, 2024, 1:53 AM

#

@cloud lantern you used CIFS, not NFS?

cloud lantern Sep 20, 2024, 1:53 AM

#

Yea, I used CIFS

#

I am copying your exact setup

pure vigil Sep 20, 2024, 1:53 AM

#

Blah

cloud lantern Sep 20, 2024, 1:56 AM

#

clear wadi Can you get all the raw settings and compare them or something like that?

I have no idea, maybe?

pure vigil Sep 20, 2024, 1:58 AM

#

If the issue is the SMB server I don’t get why I can overwrite, open, delete, those same files (like in other contexts)

#

🤷‍♂️

cloud lantern Sep 20, 2024, 1:59 AM

#

pure vigil If the issue is the SMB server I don’t get why I can overwrite, open, delete, t...

The other contexts may not open the file in write mode

clear wadi Sep 20, 2024, 2:00 AM

#

Man this is so annoying

#

I am so unsatisfied

cloud lantern Sep 20, 2024, 2:01 AM

#

https://github.com/nodejs/node/issues/41093

#

https://github.com/libuv/libuv/pull/3380

torn stormBOT Sep 20, 2024, 2:01 AM

#

cloud lantern https://github.com/libuv/libuv/pull/3380

[Pull Request] win: fix EPERM when trying to write a hidden file (libuv/libuv#3380)

cloud lantern Sep 20, 2024, 2:01 AM

#

I found the issues

clear wadi Sep 20, 2024, 2:03 AM

#

Is it because the filesystem is still "Windows"

#

Even if you run code in wsl?

cloud lantern Sep 20, 2024, 2:04 AM

#

Honestly I am at a loss

#

Why the hell can't I reproduce it man

clear wadi Sep 20, 2024, 2:04 AM

#

On Windows right

cloud lantern Sep 20, 2024, 2:04 AM

#

I am running in WSL, from command prompt, docker compose is the exact same

clear wadi Sep 20, 2024, 2:04 AM

#

And it is marked as hidden?

cloud lantern Sep 20, 2024, 2:04 AM

#

I am connecting to truenas core

pure vigil Sep 20, 2024, 2:04 AM

#

cloud lantern https://github.com/libuv/libuv/pull/3380

These are exactly the issue too

clear wadi Sep 20, 2024, 2:04 AM

#

And override works?

cloud lantern Sep 20, 2024, 2:04 AM

#

File shows as hidden

#

Yep overwrite works, the server starts just fine

clear wadi Sep 20, 2024, 2:05 AM

#

Can you open with the w flag explicitly?

cloud lantern Sep 20, 2024, 2:05 AM

#

Well node does that

#

So yes

#

#

Totally fine

#

What version of docker desktop are you using @pure vigil ?

pure vigil Sep 20, 2024, 2:08 AM

#

Latest 4.34.2

cloud lantern Sep 20, 2024, 2:08 AM

#

Can you run wsl -l -v?

pure vigil Sep 20, 2024, 2:10 AM

#

cloud lantern Sep 20, 2024, 2:10 AM

#

The only difference I can see is you have docker-desktop-data, which is something that was removed in docker version 4.30.0

#

But if you had a previous version installed before, it won't remove it, it'll just keep using it

#

But I don't see that causing this issue

clear wadi Sep 20, 2024, 2:13 AM

#

On Windows, opening an existing hidden file using the 'w' flag (either through fs.open(), fs.writeFile(), or fsPromises.open()) will fail with EPERM. Existing hidden files can be opened for writing with the 'r+' flag.

#

It is in the fs node docs actually

#

https://nodejs.org/docs/latest/api/fs.html#file-system-flags

#

Zack needs to go to bed because I am going to bed and he is like 5 hours ahead of me lol

pure vigil Sep 20, 2024, 2:16 AM

#

lol yah dude. You’ve been at this for a while. If there’s a fix or a workaround immich can implement that would be awesome. But either way, I really appreciate you guys spending so much time on this. And after reading that issue and in the node docs… super goofy. I don’t know why you can’t reproduce it though

cloud lantern Sep 20, 2024, 2:18 AM

#

clear wadi https://nodejs.org/docs/latest/api/fs.html#file-system-flags

I mean, writeFile takes flags

#

So we could manually handle this

clear wadi Sep 20, 2024, 2:18 AM

#

It looks like it has to do with the actual underlying filesystem call, on the windows side

cloud lantern Sep 20, 2024, 2:18 AM

#

But just seems like pain

clear wadi Sep 20, 2024, 2:18 AM

#

cloud lantern But just seems like pain

Agreed.

cloud lantern Sep 20, 2024, 2:18 AM

#

I'd rather just never touch dotfiles ever again :))

clear wadi Sep 20, 2024, 2:18 AM

#

I think we know why, but best to leave dot files behind at this point

cloud lantern Sep 20, 2024, 2:18 AM

#

#

lol

clear wadi Sep 20, 2024, 2:18 AM

#

Dot files on Windows are cursed

cloud lantern Sep 20, 2024, 2:18 AM

#

It just works man...

clear wadi Sep 20, 2024, 2:19 AM

#

I think fs.write is a lower level api

#

You use it with fs.open and I assume specifying the wrong flags would be a problem.

#

Unless this isn't a "real" windows partition or something

#

I have a Windows machine I'll boot to and test tomorrow I think

cloud lantern Sep 20, 2024, 2:21 AM

#

You can specify the flags with fs.writeFile

#

But yes, r+ only works if the file exists

#

Whereas w works even if it doesn't

clear wadi Sep 20, 2024, 2:21 AM

#

I see. The problem seems to be fs open with the wrong flags causes Windows to throw an error. If it doesn't for you I don't know what to say lol

cloud lantern Sep 20, 2024, 2:22 AM

#

Yea, I agree that's the problem

clear wadi Sep 20, 2024, 2:22 AM

#

Maybe they fixed it in Windows 11 🤣🤣🤣

cloud lantern Sep 20, 2024, 2:22 AM

#

But I am baffled that it isn't happening here 😛

cloud lantern Sep 20, 2024, 2:49 AM

#

Btw, @pure vigil I think if you go into your samba share settings and add hide dot files = no into the Auxillary parameters

#

Then turn the SMB service off and on again

#

That'll fix the issue for you until we deploy a patch

pure vigil Sep 20, 2024, 2:51 AM

#

Oh I will try this tonight. Thank you

cloud lantern Sep 20, 2024, 2:52 AM

#

I'm done trying to reproduce this though, at least for now 🤣

#

I even downgraded my docker desktop so I had the two different WSL instances

#

I'm just deemed worthy of being allowed to write to hidden files obviously...

pure vigil Sep 20, 2024, 2:55 AM

#

Yep

#

That worked

cloud lantern Sep 20, 2024, 2:55 AM

#

Neat

#

At least we can get you up and running again 😛

pure vigil Sep 20, 2024, 2:56 AM

#

#

Thank you so much

cloud lantern Sep 20, 2024, 2:56 AM

#

And it's only 4AM, look at that!

#

🤣

pure vigil Sep 20, 2024, 2:56 AM

#

LOL oh man

#

Go to bed. But seriously, thank you

cloud lantern Sep 20, 2024, 2:56 AM

#

@clear wadi we'll see who wakes up first in the morning 😛

ebon fulcrum Sep 20, 2024, 3:57 AM

#

cloud lantern https://github.com/nodejs/node/issues/41093

The linked issue cracks me up https://github.com/oven-sh/bun/issues/6820

#

Bun tried to warn us, but it’s too late now 😔

mighty needle Sep 20, 2024, 5:59 AM

#

Well this was a wild ride to wake up to

cloud lantern Sep 20, 2024, 12:34 PM

#

Hey @pure vigil, whenever you're around, if you'd be willing to revert the change to your SMB config from last night and upgrade to a test build for us, it'd be much appreciated 🙂

#

So if you could remove that SMB setting around the hidden dot files, restart SMB service, then verify your instance doesn't reboot correctly anymore.

#

Then, if you could set your immich-server to use this image ghcr.io/immich-app/immich-server:pr-12812 and docker compose down, docker compose pull then docker compose up -d and see if the server starts correctly, that'd be great! 🙂

#

You're like, the only reliably reproduction case we have for this issue 🤣

pure vigil Sep 20, 2024, 3:23 PM

#

Yep. I can do this in a couple hours. (Maybe less. I have one thing for work I need to take care of first but I will do this and report back. Won’t take long to check.)

cloud lantern Sep 20, 2024, 3:28 PM

#

Perfect! Thanks a bunch 🙂

pure vigil Sep 20, 2024, 3:55 PM

#

reverted the SMB config. confirmed broken again. going to try the new build.

#

looks good to me

#

#

yeah; it's working

mighty needle Sep 20, 2024, 3:59 PM

#

What changed 👀

bronze vessel Sep 20, 2024, 4:00 PM

#

@cloud lantern great news!

cloud lantern Sep 20, 2024, 4:01 PM

#

Excellent!

pure vigil Sep 20, 2024, 4:01 PM

#

assuming fix was changing the test file name?

cloud lantern Sep 20, 2024, 4:01 PM

#

It's still a hidden file actually

#

We just changed the way we write files in the server 🙂

#

Thanks so much for testing, we'll probably roll this out early next week

pure vigil Sep 20, 2024, 4:01 PM

#

all good. thank you for fixing!

cloud lantern Sep 20, 2024, 4:01 PM

#

You should be good to stay on this release until then, then update to that version when it's out

#

Though, this is pre-release

pure vigil Sep 20, 2024, 4:01 PM

#

appreciate you guys.

cloud lantern Sep 20, 2024, 4:02 PM

#

So you can also rollback to 1.115.0 if you'd prefer 😛

pure vigil Sep 20, 2024, 4:02 PM

#

nah. i'll stay on it. if something breaks i'll let you know

cloud lantern Sep 20, 2024, 4:02 PM

#

Everything should be compatible with doing that

#

Cool, sounds good 🙂

pure vigil Sep 20, 2024, 4:02 PM

#

thanks guys

#Latest update causing reboot loop

References

Checklist