Broken Immich install with PGSQL exiting | Immich | Page 1

drifting lava Apr 15, 2025, 11:51 AM

#

Hello there,

I've been running into issues with my Immich install running on a Proxmox VM with Ubuntu 22.04 LTS in Docker.
It used to work as non-root but stopped working, complaining about missing pg_notify folder.
It seems the errors are related to permissions for the postgresql folder (mounted with bind mounts in a folder on another local drive), with missing permissions, even though I switched to running the PG container without UID/GID options to unblock the situation.

At first I was getting error in the docker logs about missing pg_notify folder. So I manually created said folder under the PG folder, set immich to run as root to avoid permission issues, but it's still not starting properly.
In the PG logs, I can now find errors, still related to missing folders. Everytime I create one, there's a new one popping up. Ex : pg_tblspc and pg_replslot then pg_twophase

Could you please help me understand what I'm doing wrong ? I feel like using bind mounts was a mistake and I should have used docker volumes :/
Thanks in advance for your help !

chrome heathBOT Apr 15, 2025, 11:52 AM

#

:wave: Hey @drifting lava,

Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich immich .

References

Container Logs: docker compose logs docs
Container Status: docker ps -a docs
Reverse Proxy: https://immich.app/docs/administration/reverse-proxy
Code Formatting https://support.discord.com/hc/en-us/articles/210298617-Markdown-Text-101-Chat-Formatting-Bold-Italic-Underline#h_01GY0DAKGXDEHE263BCAYEGFJA

#

Checklist

I have...

:blue_square: verified I'm on the latest release(note that mobile app releases may take some time).
:blue_square: read applicable release notes.
:blue_square: reviewed the FAQs for known issues.
:blue_square: reviewed Github for known issues.
:blue_square: tried accessing Immich via local ip (without a custom reverse proxy).
:blue_square: uploaded the relevant information (see below).
:blue_square: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable

(an item can be marked as "complete" by reacting with the appropriate number)

Information

In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:

Your docker-compose.yml and .env files.
Logs from all the containers and their status (see above).
All the troubleshooting steps you've tried so far.
Any recent changes you've made to Immich or your system.
Details about your system (both software/OS and hardware).
Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
The version of the Immich server, mobile app, and other relevant pieces.
Any other information that you think might be relevant.

Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)

If this ticket can be closed you can use the /close command, and re-open it later if needed.

drifting lava Apr 15, 2025, 11:52 AM

#

Ex :

2025-04-15 11:42:48.107 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2025-04-15 11:42:48.107 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2025-04-15 11:42:48.117 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2025-04-15 11:42:48.138 UTC [28] LOG:  database system shutdown was interrupted; last known up at 2025-04-14 13:32:55 UTC
[2025-04-15T11:42:48Z INFO  service::utils::clean] Find directory "pg_vectors/indexes/17575".
[2025-04-15T11:42:48Z INFO  service::utils::clean] Find directory "pg_vectors/indexes/21880".
[2025-04-15T11:42:48Z INFO  service::utils::clean] Find directory "pg_vectors/indexes/17575/segments/30674d0d-9e88-4289-a87d-ae0977b0f129".
[2025-04-15T11:42:48Z INFO  service::utils::clean] Find directory "pg_vectors/indexes/21880/segments/c53a3bfe-9273-4a7e-a906-a3f61f7fd797".
2025-04-15 11:42:48.672 UTC [28] FATAL:  could not open directory "pg_twophase": No such file or directory
2025-04-15 11:42:48.682 UTC [1] LOG:  startup process (PID 28) exited with exit code 1
2025-04-15 11:42:48.682 UTC [1] LOG:  aborting startup due to startup process failure
2025-04-15 11:42:48.689 UTC [1] LOG:  database system is shut down```

formal wave Apr 15, 2025, 11:53 AM

#

What do you mean by a bind mount to a local drive? Local how

drifting lava Apr 15, 2025, 11:53 AM

#

📎 message.txt

#

It's a bind mount pointing to a folder that is on a secondary drive, not the system drive, mounted using fstab

#

but it's not a network drive, it's still in the same computer

formal wave Apr 15, 2025, 11:54 AM

#

Can I see the fstab line in the VM?

drifting lava Apr 15, 2025, 11:55 AM

#

sure : /dev/disk/by-uuid/84d18d66-fc9f-430f-a267-c0550c39f567 /mnt/data ext4 defaults,noatime,nodiratime 0 2

formal wave Apr 15, 2025, 11:55 AM

#

Its not clear to me why you would ever have permission issues to begin with

#

Do you have some weird script or something that’s changing disk permissions?

#

Or something that would delete empty folders? Any kind of space saver?

drifting lava Apr 15, 2025, 11:57 AM

#

not that I know of, I used to run with different permissions though, with UID/GID set on all containers (and it's been running like this for month)s, but since I started debugging this, I removed all settings of UID/GID to run as root, and avoid permissions problems

formal wave Apr 15, 2025, 11:57 AM

#

I’m not sure if that’s a good thing in this case. Because I think Postgres might drop to a different user regardless

#

So by changing to root you’re really retuning to the default PG user - 999?

drifting lava Apr 15, 2025, 11:58 AM

#

ah

#

so I should try switching back to forcing UID/GID ?

formal wave Apr 15, 2025, 11:58 AM

#

Based on some quick googling seems all these folders are often empty

#

So I suspect something is deleting empty folders on your system.

#

Probably, and you can chown -R the whole folder to the UID
How are you setting UID in docker?

drifting lava Apr 15, 2025, 12:00 PM

#

' user: ${UID}:${GID}' line for each container

#

and UID and GID are set in a .env

#

let me try that

formal wave Apr 15, 2025, 12:01 PM

#

Should work unless UID is like a docker special variable or something

#

I’d try using a more unique variable name

drifting lava Apr 15, 2025, 12:02 PM

#

770 for permissions ?

formal wave Apr 15, 2025, 12:03 PM

#

Shouldn’t have to change that, I would leave those as is

drifting lava Apr 15, 2025, 12:06 PM

#

:/ still getting 2025-04-15 12:05:15.253 UTC [15] FATAL: could not open directory "pg_twophase": No such file or directory in PG logs and seeing the PG container exiting 1

#

I'll try creating it

formal wave Apr 15, 2025, 12:09 PM

#

Yeah if the folder is absent on disk the owner/group won’t fix that

#

It might be better to restore from a DB backup

#

You have to figure out why your filesystem is losing folders though

drifting lava Apr 15, 2025, 12:16 PM

#

I'm not sure tbh, I've cleaned some docker volumes, but it shouldn't be affecting the bind mounts

#

or might have been power loss at unexpected times

#

I'll try following the db restore process, thanks !

formal wave Apr 15, 2025, 12:27 PM

#

I’d be surprised if power loss only affected unused empty folders

drifting lava Apr 15, 2025, 12:28 PM

#

true

#

quick question, when restoring with

## Uncomment the next line and replace DB_DATA_LOCATION with your Postgres path to permanently reset the Postgres database
# rm -rf DB_DATA_LOCATION # CAUTION! Deletes all Immich data to start from scratch
docker compose pull             # Update to latest version of Immich (if desired)
docker compose create           # Create Docker containers for Immich apps without running them
docker start immich_postgres    # Start Postgres server
sleep 10                        # Wait for Postgres server to start up
# Check the database user if you deviated from the default
gunzip --stdout "/path/to/backup/dump.sql.gz" \
| sed "s/SELECT pg_catalog.set_config('search_path', '', false);/SELECT pg_catalog.set_config('search_path', 'public, pg_catalog', true);/g" \
| docker exec -i immich_postgres psql --dbname=postgres --username=<DB_USERNAME>  # Restore Backup
docker compose up -d            # Start remainder of Immich apps```

I'm assuming dbname needs to match my db name from my .env. Shouldn't I also set --password=<DB_PASSWORD> ?

formal wave Apr 15, 2025, 12:41 PM

#

No

#

Dbname doesn’t need to change either

drifting lava Apr 15, 2025, 12:42 PM

#

ok !

#

thanks !

#

err, running line by line the restore script

#

## Uncomment the next line and replace DB_DATA_LOCATION with your Postgres path to permanently reset the Postgres database
# rm -rf DB_DATA_LOCATION # CAUTION! Deletes all Immich data to start from scratch
docker compose pull             # Update to latest version of Immich (if desired)
docker compose create           # Create Docker containers for Immich apps without running them
docker start immich_postgres    # Start Postgres server
sleep 10                        # Wait for Postgres server to start up```

#

at this point, I check the logs

#

and see

#

immich_postgres  | 2025-04-15T12:47:15.969784971Z chmod: changing permissions of '/var/run/postgresql': Operation not permitted
immich_postgres  | 2025-04-15T12:47:16.072638789Z The files belonging to this database system will be owned by user "postgres".
immich_postgres  | 2025-04-15T12:47:16.072768965Z This user must also own the server process.
immich_postgres  | 2025-04-15T12:47:16.072799004Z 
immich_postgres  | 2025-04-15T12:47:16.072821433Z The database cluster will be initialized with locale "en_US.utf8".
immich_postgres  | 2025-04-15T12:47:16.072838846Z The default database encoding has accordingly been set to "UTF8".
immich_postgres  | 2025-04-15T12:47:16.072852012Z The default text search configuration will be set to "english".
immich_postgres  | 2025-04-15T12:47:16.072864908Z 
immich_postgres  | 2025-04-15T12:47:16.072876953Z Data page checksums are enabled.
immich_postgres  | 2025-04-15T12:47:16.072889392Z 
immich_postgres  | 2025-04-15T12:47:16.072977185Z initdb: error: could not change permissions of directory "/var/lib/postgresql/data": Operation not permitted```

#

etc...

#

so I'm basically still getting permissions errors even though I wiped the DB folder :/, and have the DB container set to use the immich user

#

oh well, looks like the PG folder is created by root :/

#

even though I have user: ${UID}:${GID} set on all containers in my compose file

formal wave Apr 15, 2025, 12:59 PM

#

If you’re using a bind mount in docker with a non root user you usually need to crate the folder and chown it before first startup.

#

That’s just a docker thing. Maybe root would be easier for you. Non root is trickier

drifting lava Apr 15, 2025, 1:02 PM

#

damn it, I had forgotten I had done that the first time I set it up

#

some progress, looks like PG is restored, thanks for that !

#

but now it's Immich machine learning's time to crash

#

with what looks like more permission issues

formal wave Apr 15, 2025, 1:06 PM

#

😬

drifting lava Apr 15, 2025, 1:07 PM

#

would I have avoided all this pain if I had gone with docker volumes instead of bind mounts ?

formal wave Apr 15, 2025, 1:08 PM

#

Not the empty folders being gone, no. That’s just an issue with your system.

drifting lava Apr 15, 2025, 1:08 PM

#

yes, but the permission issues ?

formal wave Apr 15, 2025, 1:08 PM

#

I think non root is the bigger issue for you Than the docker volumes

#

Non root may be easier in them but still has caveats

drifting lava Apr 15, 2025, 1:09 PM

#

yes, I agree non-root isn't that fun, but I work in cybersecurity, and have great fears of running as root imprinted in my bones due to this ^^

formal wave Apr 15, 2025, 1:10 PM

#

Haha I mean that’s fair

#

Docker volumes probably would be easier but IMO this kind of stuff needs to be second nature if you run non root because you run into these issues often in my experience

#

All it saves you is the initial chown

drifting lava Apr 15, 2025, 5:10 PM

#

By chance, would you happen to know how to fix this ? I've wiped the bind mounts for the ML containers, and replaced them with docker volumes; from scratch, I started running the stack with non-root uid/gid, but I'm still getting that error when starting the stack

📎 message.txt

#

for my sanity s sake I'm going to try running the immich-ml as root to see if it solves this

formal wave Apr 15, 2025, 5:12 PM

#

Looks like more permissions issues

#

Can’t create tmp folders etc

drifting lava Apr 15, 2025, 5:21 PM

#

yup, but what I can't understand is that I'm mounting the volumes listed here : https://immich.app/docs/FAQ/#how-can-i-run-immich-as-a-non-root-user.
Extract from my compose :

    container_name: immich_machine_learning
    user: ${IMMICHUID}:${IMMICHGID}
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
      - ml-cache:/.cache
      - ml-config:/.config
    env_file:
      - .env
    restart: always
    healthcheck:
      disable: false```

#

I'm trying with no UID/GID defined for root

#

it's working with no UID/GID defined

#

yay for database backups I guess

#

I'd love to understand why immich-ml is not playing nice with defined UID/GID and those volumes config

formal wave Apr 15, 2025, 5:36 PM

#

What OS are you on

drifting lava Apr 15, 2025, 6:34 PM

#

Ubuntu 22.04 LTS

formal wave Apr 15, 2025, 6:34 PM

#

This isn’t a snap docker install is it?

drifting lava Apr 15, 2025, 6:35 PM

#

no, it's installed using apt

formal wave Apr 15, 2025, 6:36 PM

#

Sorry I think your system is just cursed lol

drifting lava Apr 15, 2025, 6:38 PM

#

it's super weird, I only use this vm as a docker host, so it's not like I've messed with tons of stuff on it

#

At some point I'll set up some iac config so that I can tear it down and build it back up, and will see if I'm encountering the same issue on the new box

#

thx a lot for the help !

#Broken Immich install with PGSQL exiting

References

Checklist

Information