Plan to migrate to local DB folder from docker volume on portainer | Immich | Page 1

wispy vessel May 17, 2024, 5:34 AM

#

Hello! I am currently on v0.99 of Immich and I am planning to update to the latest with a new docker compose file and opting in to the change to migrate my PG database to a local folder instead of keep it on a docker volume. I am assuming this will make my installation much harder to break (since presumably I could reinstall immich in its entirety without even having to make a backup since all local data is now separate relative to the docker container, if I understand it properly, please correct me if I am wrong), so I have made a pg_dumpall backup and am now about to try to perform the migration. I was wondering then if there was any guide on how to perform the migration to the new DB system like how to mount a local directory to an arbitrary container in portainer? I've been looking into this for a while now and am getting a bit lost following the instructions in the release notes.

coarse sierraBOT May 17, 2024, 5:34 AM

#

:wave: Hey @wispy vessel,

Thanks for reaching out to us. Please follow the recommended actions below; this will help us be more effective in our support effort and leave more time for building Immich immich .

References

Container Logs: docker compose logs docs
Container Status: docker compose ps docs
Reverse Proxy: https://immich.app/docs/administration/reverse-proxy

Checklist

:blue_square: I have verified I'm on the latest release(note that mobile app releases may take some time).
:ballot_box_with_check: I have read applicable release notes.
:ballot_box_with_check: I have reviewed the FAQs for known issues.
:ballot_box_with_check: I have reviewed Github for known issues.
:ballot_box_with_check: I have tried accessing Immich via local ip (without a custom reverse proxy).
:ballot_box_with_check: I have uploaded the relevant logs, docker compose, and .env files using the buttons below or the /upload command.
:ballot_box_with_check: I have tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable

(an item can be marked as "complete" by reacting with the appropriate number)

If this ticket can be closed you can use the /close command, and re-open it later if needed.

#

@wispy vessel uploaded

📎 docker-compose.yml

wispy vessel May 17, 2024, 5:37 AM

#

(upload of .env)

native skiff May 17, 2024, 12:05 PM

#

I downed all but postgres, made dump.
Down compose completely
Change compose to hostmount bind
Only create the compose file, not starting any containers.
Only start postgres
See Restore in docs how to restore database.
Start the rest.

wispy vessel May 17, 2024, 5:23 PM

#

Will that approach allow me to move my postgres data to an external folder? How would I gain access to the volume database data to move it out of the container with that strategy?

native skiff May 17, 2024, 7:13 PM

#

In step 1, i make a backup of the database.
But you can also mount the volume to a different container.
read the docs on backup and restore, and also the changes in the patch notes.

wispy vessel May 17, 2024, 7:31 PM

#

Oh so I basically save my upload_location and pg_dumpall, wipe my current immich install, and use the restore commands to create a new running Immich instance with the same upload_location?

#

How do these commands play into the portainer stack deployment?

#

like should I just run these commands in a terminal on the machine running immich, and portainer will automatically recognize the stack?

native skiff May 17, 2024, 7:32 PM

#

https://immich.app/docs/administration/backup-and-restore/

#

Patch note:

I want to migrate my docker volume to a local folder

Unfortunately there isn't a "proper" way to export a docker volume.
The recommended method is to mount the volume and the directory (you want to copy your data to) to an arbitrary container, get a shell inside that container and copy the folder manually.

Caution

Take backups before attempting this. Especially make sure you have a current database dump (pg_dump)

Warning

Do not use a directory under /mnt for the postgres location if you are using WSL.
Generally (on all operating systems) we recommend against using a network share for your database location. This is bound to break and cause all sorts of weird issues.

If you would like to opt-in to this change, there is an additional environment variable in the .env file as well as a modification in your existing docker-compose.yml file.

wispy vessel May 17, 2024, 7:33 PM

#

Yeah I raed that from the patch notes in v.102

#

The part I was having trouble with is how do I make an arbitrary container in portainer and mount a local directory as a volume

#

I tried making just a random nginx container and starting it up for example but I'm not sure how to bind a local folder to that container and have access to it if I get a shell into that container

native skiff May 17, 2024, 7:34 PM

#

I dont use portainer. I used to try it once. But basically, you need the database somehow. Either via pgdump (a backup) or via mounting the database docker volume to a different container, then copying the data out of the container to your host.

wispy vessel May 17, 2024, 7:35 PM

#

Hmmm so if I have a pg_dump

native skiff May 17, 2024, 7:35 PM

#

I have less experiernce with that too. But i know there is a docker command to copy files out of a container to your host.

wispy vessel May 17, 2024, 7:35 PM

#

Can I simply put that into the db_location folder and Immich will work with that filesystem?

#

Oh is there

#

hmm I thought it can do that for files but not volumes though

native skiff May 17, 2024, 7:36 PM

#

Follow those commands you saw. Dont change anything else. The upload location remains the same. Only backup the database, update docker compose file, and restore

native skiff May 17, 2024, 7:37 PM

#

wispy vessel hmm I thought it can do that for files but not volumes though

Ye i wouldnt know either right now honestly. But if a volume is mounted in a container. It must be available in the container right?

#

pgdata:/var/lib/postgresql/data

wispy vessel May 17, 2024, 7:37 PM

#

Ohhhhh smart so I can just mount the volume and then try docker copying to the local amchine

#

smart smart

#

I'll try that

wispy vessel May 17, 2024, 7:38 PM

#

native skiff - pgdata:/var/lib/postgresql/data

ooh ty ty

native skiff May 17, 2024, 7:38 PM

#

I think thats what they are saying right? But dont take my word on it

wispy vessel May 17, 2024, 7:38 PM

#

Yeah I mean if it doesn't work its just a docker copy so nothing should break

#

I can go back to the drawing board if this doesn't work

native skiff May 17, 2024, 7:38 PM

#

Make sure u use a different container. Not immich

wispy vessel May 17, 2024, 7:38 PM

#

Ooh okay I'll make a random nginx container

native skiff May 17, 2024, 7:38 PM

#

You dont want a running instance and copy database files. Chances of corruption

wispy vessel May 17, 2024, 7:39 PM

#

Do you happen to know if I copy just everything in "pgdata:/var/lib/postgresql/data" into the local folder "db_location" cirectly?

#

like using no subfolders so everything in data/ is now in db_location/

wispy vessel May 17, 2024, 7:39 PM

#

native skiff You dont want a running instance and copy database files. Chances of corruption

oh crap I wa about to mess that up

#

so I stop the immich stack first

native skiff May 17, 2024, 7:40 PM

#

Yes

wispy vessel May 17, 2024, 7:40 PM

#

good catch ty

native skiff May 17, 2024, 7:40 PM

#

The recommended method is to mount the volume and the directory (you want to copy your data to) to an arbitrary container, get a shell inside that container and copy the folder manually.

#

So that is the recommended way.

wispy vessel May 17, 2024, 7:40 PM

#

Gotcha I'll try that rq and let you know how it goes

#

ty for the help so far btw

native skiff May 17, 2024, 7:41 PM

#

Mount bind the new location for your database also. Copy it inside the container

#

Make a database dump first!

wispy vessel May 17, 2024, 7:41 PM

#

Yesss database dump always

#

pg_dumpall all day every day

native skiff May 17, 2024, 7:41 PM

#

Haha

wispy vessel May 17, 2024, 7:41 PM

#

native skiff Mount bind the new location for your database also. Copy it inside the container

Mount bind the local folder to the arbitrary container right?

native skiff May 17, 2024, 7:41 PM

#

Yes

wispy vessel May 17, 2024, 7:42 PM

#

okok I'll see if I can not break anything

wispy vessel May 17, 2024, 8:34 PM

#

Okay so I think I have made some good progress and just want to confirm something

#

I have copied over the volume to a local folder called ImmichDatabase

#

is there any folder structure that I should re-create under the local database directory?

#

This is what the DB_DATA_LOCATION folder looks like

#

#

Do you know if yours follows a similar structure?

#

If so, then I can go ahead and start the stack to see if it works

#

I updated the compose file with the new db stuff and added the folder ImmichDatabase as the DB_DATA_LOCATION in the .env

wispy vessel May 18, 2024, 1:15 AM

#

Alright update, I got it to work and I think I am now in the new database structure!

#Plan to migrate to local DB folder from docker volume on portainer

References

Checklist