#Updating from v1.106.4 to v1.107.0 breaks something

1 messages · Page 1 of 1 (latest)

willow stream
#

Seems like updating from v1.106.4 to v1.107.0 breaks something, anyone else see this?

Detected CPU Cores: 16
Starting api worker
Starting microservices worker
[Nest] 7  - 07/02/2024, 4:06:07 PM     LOG [Microservices:EventRepository] Initialized websocket server
Migration "AddFaceSearchRelation1718486162779" failed, error: type "vector" does not exist
latent breachBOT
#

:wave: Hey @willow stream,

Thanks for reaching out to us. Please follow the recommended actions below; this will help us be more effective in our support effort and leave more time for building Immich immich.

References

Checklist

  1. :blue_square: I have verified I'm on the latest release(note that mobile app releases may take some time).
  2. :blue_square: I have read applicable release notes.
  3. :blue_square: I have reviewed the FAQs for known issues.
  4. :blue_square: I have reviewed Github for known issues.
  5. :blue_square: I have tried accessing Immich via local ip (without a custom reverse proxy).
  6. :blue_square: I have uploaded the relevant logs, docker compose, and .env files using the buttons below or the /upload command.
  7. :blue_square: I have tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable

(an item can be marked as "complete" by reacting with the appropriate number)

If this ticket can be closed you can use the /close command, and re-open it later if needed.

valid osprey
#

Are you using stock compose file? Can you provide the information the bot asked for?

willow stream
#

I don't use docker-compose

lone hearth
#

you can post your compose and env, what version are you coming from? does the web ui still work?

valid osprey
#

Can you post your setup?

willow stream
#

the only thing I'm changing is the image tag at the top, which gives me the error mentioned

valid osprey
willow stream
#

Yeah the image I'm running for pg is image = "tensorchord/pgvecto-rs:pg14-v0.2.0"

valid osprey
#

Let's bring in the wizard, @rugged talon do you have any thoughts on the vector not found message?

lone hearth
#

can we see the setup for postgres.service.consul?

willow stream
lone hearth
#

how long have you been running this setup?

#

I'm wondering if you need to add command: ["postgres", "-c" ,"shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"], which we now specify in the docker-compose.yml. Not sure why it would cause an issue starting with this version, though

#

can you connect to the DB console?

willow stream
#

git blame:

Tue 20 Feb 2024 08:18:15 PM CET • nomad: update postgres to pg14-v0.2.0
lone hearth
#

hm.

willow stream
#

I've followed the steps along the way through upgrades, so I've enabled whatever extensions and things have been instructed in the release notes up to this point

lone hearth
#

yeah, and this change shouldn't be required, at least with a standard docker setup.
Is it possible to run something like docker inspect with your setup? and/or connect to the PG console to inspect the extensions and table layout?

willow stream
#

yeah I can, just let me know what you want me to check for

lone hearth
#

connect to the console, type \c immich, then \l+ \d+ \dn+ \zand \dx+

#

each one is a command

willow stream
#

I know, I'm not a database super user, but I've dabbled

#
immich=# \l+
                                                                    List of databases
    Name     |  Owner   | Encoding |  Collate   |   Ctype    |   Access privileges   |  Size   | Tablespace |                Descript
ion
-------------+----------+----------+------------+------------+-----------------------+---------+------------+------------------------
--------------------
 immich      | immich   | UTF8     | en_US.utf8 | en_US.utf8 |                       | 253 MB  | pg_default |
 postgres    | postgres | UTF8     | en_US.utf8 | en_US.utf8 |                       | 8577 kB | pg_default | default administrative
connection database
 template0   | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +| 8409 kB | pg_default | unmodifiable empty data
base
             |          |          |            |            | postgres=CTc/postgres |         |            |
 template1   | postgres | UTF8     | en_US.utf8 | en_US.utf8 | =c/postgres          +| 8433 kB | pg_default | default template for ne
w databases
             |          |          |            |            | postgres=CTc/postgres |         |            |
             |          |          |            |            | postgres=CTc/postgres |         |            |
(9 rows)
#

I'm obfuscating a few of the unrelated databases listed here

lone hearth
#

ok so you're using the vecto.rs DB for other services as well?

willow stream
#

yeah, it's a shared database

#

I just switched over to using the verto.rs variant when that came up with immich, I haven't noticed any issues

lone hearth
#

ok so to be clear, how did you switch? did this start as a non-vecto.rs database that you merged into this one?

willow stream
#

yes, it was a vanilla postgres 14 image before that

lone hearth
#

and how did you change the image type?

#

did you just change the image:, or did you do a dump and restore?

valid osprey
#

Can you try run this again?

DROP EXTENSION IF EXISTS vectors;
CREATE EXTENSION vectors;
willow stream
#

don't recall as it was a while back, but I at least dumped for backup

#
immich=# DROP EXTENSION IF EXISTS vectors;
ERROR:  cannot drop extension vectors because other objects depend on it
DETAIL:  column embedding of table asset_faces depends on type vector
column embedding of table smart_search depends on type vector
HINT:  Use DROP ... CASCADE to drop the dependent objects too.
immich=# CREATE EXTENSION vectors;
ERROR:  extension "vectors" already exists
immich=#
valid osprey
#

okay

#

so you have the extension

willow stream
#

\dn+

immich=# \dn+
                          List of schemas
  Name   |  Owner   |  Access privileges   |      Description
---------+----------+----------------------+------------------------
 public  | postgres | postgres=UC/postgres+| standard public schema
         |          | =UC/postgres         |
 vectors | postgres |                      |
(2 rows)
lone hearth
#

can you try GRANT ALL ON SCHEMA vectors TO <immichusername>; ?

#

This is mine:

immich=# \dn+
                          List of schemas
  Name   |  Owner   |  Access privileges   |      Description
---------+----------+----------------------+------------------------
 public  | postgres | postgres=UC/postgres+| standard public schema
         |          | =UC/postgres         |
 vectors | postgres | immich=U/postgres   +|
         |          | postgres=UC/postgres |
(2 rows)
willow stream
#
immich=# GRANT ALL ON SCHEMA vectors TO immich;
GRANT
immich=# \dn+
                          List of schemas
  Name   |  Owner   |  Access privileges   |      Description
---------+----------+----------------------+------------------------
 public  | postgres | postgres=UC/postgres+| standard public schema
         |          | =UC/postgres         |
 vectors | postgres | postgres=UC/postgres+|
         |          | immich=UC/postgres   |
(2 rows)
#

api container:

Detected CPU Cores: 16
Starting api worker
Starting microservices worker
[Nest] 7  - 07/02/2024, 4:40:36 PM     LOG [Microservices:EventRepository] Initialized websocket server
[Nest] 7  - 07/02/2024, 4:40:36 PM    WARN [Microservices:DatabaseService] Could not run vector reindexing checks. If the extension was updated, please restart the Postgres instance.
#

guess I'll have to restart pg

#

restarted pg twice, still getting this from the api container:

Detected CPU Cores: 16
Starting api worker
Starting microservices worker
[Nest] 7  - 07/02/2024, 4:45:14 PM     LOG [Microservices:EventRepository] Initialized websocket server
[Nest] 7  - 07/02/2024, 4:45:14 PM    WARN [Microservices:DatabaseService] Could not run vector reindexing checks. If the extension was updated, please restart the Postgres instance.
lone hearth
#

can you run SELECT pgvectors_upgrade();?

willow stream
#
immich=# SELECT pgvectors_upgrade();
WARNING:  pgvecto.rs is upgraded. Restart PostgreSQL to take effects.
 pgvectors_upgrade
-------------------

(1 row)
#

hmm, why is it asking to restart again in the console

valid osprey
#

yeah let's try to restart your postgres database, then restart Immich stack

willow stream
#

it still says the same after another restart

lone hearth
#

can we see the PG logs?

willow stream
lone hearth
#

2024-07-02 18:48:31.140 CEST [53] ERROR: permission denied for view pg_vector_index_stat

willow stream
#

yeah

lone hearth
#

you're going to want to become intimately familiar with this page with your setup haha

#

for anyone looking back, just switching out the database images is *not supported, this should have been a pg_dumpall and restore into the new database to avoid these issues

#

but it should be fixable

willow stream
#

so should I run through those steps laid out in that doc?

lone hearth
#

just the part pertaining to the current error

willow stream
#

ah

#

I was skimming too fast

#

restarting pg again...

#

starting immich...

#

hrmm

#

still the same

#
Detected CPU Cores: 16
Starting api worker
Starting microservices worker
[Nest] 7  - 07/02/2024, 4:55:21 PM     LOG [Microservices:EventRepository] Initialized websocket server
[Nest] 19  - 07/02/2024, 4:55:21 PM     LOG [Api:EventRepository] Initialized websocket server
[Nest] 7  - 07/02/2024, 4:55:21 PM    WARN [Microservices:DatabaseRepository] Could not reindex index clip_index. Attempting to auto-fix.
[Nest] 7  - 07/02/2024, 4:55:23 PM    WARN [Microservices:DatabaseRepository] Could not reindex index face_index. Attempting to auto-fix.
[Nest] 7  - 07/02/2024, 4:55:23 PM    WARN [Microservices:DatabaseService] Could not run vector reindexing checks. If the extension was updated, please restart the Postgres instance.
lone hearth
#

PG logs?

#

let's see \z in the immich DB again

willow stream
lone hearth
#

ok

#

we need to reindex some stuff

#

hang ong

#

Actually, the fix for that might be another restart of both PG then immich

willow stream
#
[2024-07-02T16:58:31Z INFO  service::utils::clean] Find directory "pg_vectors/indexes/567070".
[2024-07-02T16:58:31Z INFO  service::utils::clean] Find directory "pg_vectors/indexes/567070/segments/a5115a2d-ccc6-457c-8f76-991538d3d6ab".
2024-07-02 18:58:31.964 CEST [1] LOG:  database system is ready to accept connections
#

this was logged by pg, but immich is still saying the same about restarting pg

lone hearth
#

let's see both PG and Immich logs again please

#

we're going in the right direction

willow stream
#

api stdout

Detected CPU Cores: 16
Starting api worker
Starting microservices worker
[Nest] 7  - 07/02/2024, 5:03:25 PM     LOG [Microservices:EventRepository] Initialized websocket server
[Nest] 7  - 07/02/2024, 5:03:25 PM    WARN [Microservices:DatabaseRepository] Could not reindex index face_index. Attempting to auto-fix.
[Nest] 19  - 07/02/2024, 5:03:25 PM     LOG [Api:EventRepository] Initialized websocket server
[Nest] 7  - 07/02/2024, 5:03:25 PM    WARN [Microservices:DatabaseService] Could not run vector reindexing checks. If the extension was updated, please restart the Postgres instance.
lone hearth
#

Could you try running REINDEX INDEX clip_index and REINDEX INDEX face_index in PG?

willow stream
#

api stderr

TypeError: Cannot read properties of undefined (reading 'dimsize')
    at DatabaseRepository.getDimSize (/usr/src/app/dist/repositories/database.repository.js:162:31)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async DatabaseRepository.reindex (/usr/src/app/dist/repositories/database.repository.js:96:33)
    at async /usr/src/app/dist/services/database.service.js:126:21
    at async /usr/src/app/dist/repositories/database.repository.js:177:23
microservices worker exited with code 1
#
immich=# REINDEX INDEX clip_index;
REINDEX
immich=# REINDEX INDEX face_index;
ERROR:  internal error: entered unreachable code
lone hearth
#

also what is your backup setup before we keep going?

willow stream
#

backup in what context?

lone hearth
#

database / VM / whatever you use

willow stream
#

the database is on disk, I could copy that or just do a pg_dumpall

lone hearth
#

I would do a pg_dumpall now, because the internal error is not promising

#

we are going in the right direction but to me this indicates a more serious issue in the database

willow stream
#

I'll make a copy of the pg data folder and then dumpall, no use skipping any of it

lone hearth
#

After, that, I would try DROP INDEX face_index;

willow stream
#

oh yeah, I just remembered that I have daily pg_dumpalls set up as well, I feel a sense of pride xD

lone hearth
#

I would save the most recent one from last night as a separate file, so you have a clean start if needed and it doesn't get rotated out

willow stream
#

yeah I copied all the backups out into a separate folder

#

okay, now I've dropped the index

#

started up immich again:

Detected CPU Cores: 16
Starting api worker
Starting microservices worker
[Nest] 7  - 07/02/2024, 5:17:57 PM     LOG [Microservices:EventRepository] Initialized websocket server
[Nest] 19  - 07/02/2024, 5:17:58 PM     LOG [Api:EventRepository] Initialized websocket server
Migration "AddFaceSearchRelation1718486162779" failed, error: internal error: entered unreachable code
lone hearth
#

I'm not sure where to go from here. As summary, we added GRANT ALL ON vectors, and we gave permissions to the pg_view_index. To be honest I have no idea how this was working before now

#

Mert might have some ideas later on

willow stream
#

Yeah let's see what input he/she/they may have, I could also see the option moving forward of just running a separate pg for immich so that it can have it's own with vecto.rs, though it's quite nice to only have one as it makes backups simple

lone hearth
#

I use one giant postgres, and it works fine... the issue here is you seem to have a sort of half-installed vectors instance because you used a new binary with a non-vectors PGdata folder

#

so it was never fully installed it seems?

#

is immich a superuser in PG?

willow stream
#

don't remember, don't think so

lone hearth
#

You might try granting SU permission next. that can fix some of these issues

#

It's not necessary if setup properly, but yours is not

willow stream
#

I could do that temporarily, but I wouldn't want to keep it that way

lone hearth
#

well you could try it for now then go one step at a time

#

I am not super expecting it to fix it but it might be worth a shot

#

is this you? #10767

latent breachBOT
rugged talon
#

So you dropped face_index?

willow stream
#

No I didn't write that issue on github

lone hearth
#

uh oh

willow stream
#

seems like I'm not alone

willow stream
rugged talon
#

I would do that. Afterwards, try running this:

SET vectors.pgvector_compatibility=on;
CREATE INDEX face_index ON asset_faces
          USING hnsw (embedding vector_cosine_ops)
          WITH (ef_construction = 300, m = 16);
willow stream
#

so drop it again and then do the above?

rugged talon
#

Yes, this is just to see if creating vector indices in general causes an error

willow stream
#

all right, one sec

#
You are now connected to database "immich" as user "postgres".
immich=# drop index face_index;
ERROR:  index "face_index" does not exist
immich=# SET vectors.pgvector_compatibility=on;
SET
immich=# CREATE INDEX face_index ON asset_faces
          USING hnsw (embedding vector_cosine_ops)
          WITH (ef_construction = 300, m = 16);
ERROR:  internal error: entered unreachable code
#

it seems to do that indeed

rugged talon
#

That’s very weird. I would try dropping clip_index too and running SELECT pgvectors_upgrade(), then restarting

willow stream
#
immich=# drop index clip_index;
DROP INDEX
immich=# SELECT pgvectors_upgrade();
WARNING:  pgvecto.rs is upgraded. Restart PostgreSQL to take effects.
 pgvectors_upgrade
-------------------

(1 row)
#

see what a restart does now then

rugged talon
#

After the restart, try making the index again

willow stream
#
postgres=# \c immich
psql (14.9, server 14.10 (Debian 14.10-1.pgdg120+1))
You are now connected to database "immich" as user "postgres".
immich=# SET vectors.pgvector_compatibility=on;
SET
immich=# CREATE INDEX face_index ON asset_faces
          USING hnsw (embedding vector_cosine_ops)
          WITH (ef_construction = 300, m = 16);
ERROR:  internal error: entered unreachable code
#

^ after restarting of course

rugged talon
#

I think it’s worth making a pgvecto.rs issue for this so they can root cause it

lone hearth
#

I believe the same pgdatafolder was used for vanilla Postgres then vectors image

willow stream
#

Whoever wrote the issue on github has the same issue though, no?

lone hearth
#

Which I assume causes issues

rugged talon
#

I suppose it could. I think it’s either 1) some internal state in the extension that’s messed up or 2) somehow the vector data itself got corrupted in a way that isn’t handled well. Restoring a backup into a fresh db is worth trying

willow stream
#

I'll try restoring a backup later this week and see how that goes, I have some other things I need to attend to right now though, but the help was much appreciated

rugged talon
#

Definitely keep us posted!

willow stream
#

"later this week" -> probably sooner rather than later, I find these things too intriguing

#

I guess I'll ping you here with new developments

valid osprey
#

From the reported issues, look like the common denominator here is for users with separate database

lone hearth
valid osprey
#

Hmm

#

If you restart your database, then just run

CREATE INDEX face_index ON asset_faces
          USING hnsw (embedding vector_cosine_ops)
          WITH (ef_construction = 300, m = 16);
#

does it help?

valid osprey
#

Ah sorry missed the message

#

@rugged talon Can we vaccum the database here?

rugged talon
#

I don’t think it’d help in this case but you could try it

willow stream
#

So this is beyond my knowledge, what do you mean by vacuuming?

rugged talon
#

You can run VACUUM FULL; and it’ll basically rewrite and reindex the entire db, cleaning things up in the process

willow stream
#

sounds like fun

#

it's still the same after vacuuming

Detected CPU Cores: 16
Starting api worker
Starting microservices worker
[Nest] 7  - 07/02/2024, 6:52:49 PM     LOG [Microservices:EventRepository] Initialized websocket server
[Nest] 19  - 07/02/2024, 6:52:49 PM     LOG [Api:EventRepository] Initialized websocket server
Migration "AddFaceSearchRelation1718486162779" failed, error: internal error: entered unreachable code
rugged talon
#

Yeah wasn’t really expecting much from that

valid osprey
#

Is this due to some new mechanism in 0.2.0 Mert?

rugged talon
#

No, I think the face search migration just exposed some kind of edge case

rugged talon
#

Can you try running these commands:

ALTER TABLE asset_faces ALTER COLUMN embedding SET DATA TYPE real[];
ALTER TABLE asset_faces ALTER COLUMN embedding SET DATA TYPE vector(512);
willow stream
#

Okay, so I applied the above changes to the tables, restarted pg, and updated to 1.107.2, and now it seems to be working

rugged talon
#

That was a total shot in the dark haha. Glad it fixed your issue!

willow stream
#

All this help has been much appreciated, great troubleshooting with you guys and I'm happy to get things working of course.
Hopefully this can also be of help for others that ended up with similar errors after the last update

valid osprey
frosty marten
#

After last update on docker
From v1.106.3 to v1.107.2, I'm getting ERR_EMPTY_RESPONSE

Role back to v1.106.3 is working

lone hearth
frosty marten
#

My bad, please ignore

still wasp
kind wagon
#

I'm having issues also after updating to the 1.107 version. I'm using docker compose and I did not change anything, just run the update stack (I'm running unraid). I'm getting similar errors as tommy. How would I fix the issue in my setup?

rugged talon
#

We’ll also update the DB migration to do this so the next patch won’t have this issue

kind wagon
#

ok thanks, I'll need to look into it because I do not have much experience on how to operarte with databases

kind wagon
#

Sorry for asking again, I installed pgadmin but now I really cannot understand how to connect to the database. I try adding a new server but then I do not know where to get the info needed there

valid osprey
kind wagon
#

I would imagine that in the address I put the IP of the server or the container IP, the port I'm not really sure about

valid osprey
# kind wagon

Assuming you are using stock information.

  • Hostname: your server ip so http://ip
  • username: postgres
  • password: postgres
  • database: immich
#

ah wait, you will need to expose the database port to access it

kind wagon
#

yes the compose file is stock, I have also the immich db dumper, not sure if that is involved in the issue also

kind wagon
#

Just to make sure, the docker exec -it immich_postgres bash line goes in the compose file in the database container part?

valid osprey
kind wagon
#

ah ok good that I asked

valid osprey
#

there should be no changes from the compose file

kind wagon
#

immich=# ALTER TABLE asset_faces ALTER COLUMN embedding SET DATA TYPE real[];
ALTER TABLE
immich=# ALTER TABLE asset_faces ALTER COLUMN embedding SET DATA TYPE vector(512);
ERROR: column "embedding" of relation "asset_faces" does not exist

#

the second command gives me an error

valid osprey
#

Can you try restart the stack?

#

bring it down and up to see you are still encountering error

#

And please post your .env file if possible

kind wagon
#

now it works