Migration "AddThumbnailJobStatus1724080823160" failed | Immich | Page 1

sullen kelp Oct 7, 2024, 11:09 PM

#

I recently updated my Immich stack and saw that the immich_server container continuously is restarting.. Looking at the logs I see this error message:

api worker exited with code 1
Detected CPU Cores: 24
Starting api worker
[Nest] 17  - 10/07/2024, 10:44:22 PM     LOG [Api:EventRepository] Initialized websocket server
Migration "AddThumbnailJobStatus1724080823160" failed, error: could not read block 35 in file "base/16384/17156": read only 0 of 8192 bytes
QueryFailedError: could not read block 35 in file "base/16384/17156": read only 0 of 8192 bytes
    at PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:219:19)
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async AddThumbnailJobStatus1724080823160.up (/usr/src/app/dist/migrations/1724080823160-AddThumbnailJobStatus.js:9:9)
    at async MigrationExecutor.executePendingMigrations (/usr/src/app/node_modules/typeorm/migration/MigrationExecutor.js:225:17)
    at async DataSource.runMigrations (/usr/src/app/node_modules/typeorm/data-source/DataSource.js:265:35)
    at async DatabaseRepository.runMigrations (/usr/src/app/dist/repositories/database.repository.js:191:9)
    at async /usr/src/app/dist/services/database.service.js:95:17
    at async /usr/src/app/dist/repositories/database.repository.js:199:23 {
  query: 'UPDATE "asset_job_status" SET "previewAt" = NOW() FROM "assets" WHERE "assetId" = "assets"."id" AND "assets"."previewPath" IS NOT NULL',
  parameters: undefined,
  driverError: error: could not read block 35 in file "base/16384/17156": read only 0 of 8192 bytes
      at /usr/src/app/node_modules/pg/lib/client.js:535:17
      at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
      at async PostgresQueryRunner.query (/usr/src/app/node_modules/typeorm/driver/postgres/PostgresQueryRunner.js:184:25)
      at async AddThumbnailJobStatus1724080823160.up (/usr/src/app/dist/migrations/1724080823160-AddThumbnailJobStatus.js:9:9)
      at async MigrationExecutor.executePendingMigrations (/usr/src/app/node_modules/typeorm/migration/MigrationExecutor.js:225:17)
      at async DataSource.runMigrations (/usr/src/app/node_modules/typeorm/data-source/DataSource.js:265:35)
      at async DatabaseRepository.runMigrations (/usr/src/app/dist/repositories/database.repository.js:191:9)
      at async /usr/src/app/dist/services/database.service.js:95:17
      at async /usr/src/app/dist/repositories/database.repository.js:199:23 {
    length: 124,
    severity: 'ERROR',
    code: 'XX001',
    detail: undefined,
    hint: undefined,
    position: undefined,
    internalPosition: undefined,
    internalQuery: undefined,
    where: undefined,
    schema: undefined,
    table: undefined,
    column: undefined,
    dataType: undefined,
    constraint: undefined,
    file: 'md.c',
    line: '685',
    routine: 'mdread'
  },
  length: 124,
  severity: 'ERROR',
  code: 'XX001',
  detail: undefined,
  hint: undefined,
  position: undefined,
  internalPosition: undefined,
  internalQuery: undefined,
  where: undefined,
  schema: undefined,
  table: undefined,
  column: undefined,
  dataType: undefined,
  constraint: undefined,
  file: 'md.c',
  line: '685',
  routine: 'mdread'
}
api worker exited with code 1

Everything worked fine on the last version and I checked the up-to-date docker-compose.yml and everything is correct on mine. Was hoping this is just something I overlooked on the update or a common issue but I couldn't see anything related in my search. I'll include my docker-compose and .env below, any help is appreciated, thanks!

wet spearBOT Oct 7, 2024, 11:09 PM

#

:wave: Hey @sullen kelp,

Thanks for reaching out to us. Please follow the recommended actions below; this will help us be more effective in our support effort and leave more time for building Immich immich .

References

Container Logs: docker compose logs docs
Container Status: docker compose ps docs
Reverse Proxy: https://immich.app/docs/administration/reverse-proxy

Checklist

:ballot_box_with_check: I have verified I'm on the latest release(note that mobile app releases may take some time).
:ballot_box_with_check: I have read applicable release notes.
:ballot_box_with_check: I have reviewed the FAQs for known issues.
:ballot_box_with_check: I have reviewed Github for known issues.
:blue_square: I have tried accessing Immich via local ip (without a custom reverse proxy).
:ballot_box_with_check: I have uploaded the relevant logs, docker compose, and .env files, making sure to use code formatting.
:blue_square: I have tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable

(an item can be marked as "complete" by reacting with the appropriate number)

If this ticket can be closed you can use the /close command, and re-open it later if needed.

sullen kelp Oct 7, 2024, 11:11 PM

#

Here is my docker-compose.yml:

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: ['start.sh', 'immich']
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - stack.env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - stack.env
    restart: always
    healthcheck:
      disable: false

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:2d1463258f2764328496376f5d965f20c6a67f66ea2b06dc42af351f75248792
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    command: ["postgres", "-c" ,"shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"]

    restart: always
    
volumes:
  model-cache:

and my .env:

UPLOAD_LOCATION=/srv/dev-disk-by-uuid-bd30c33b-aedb-4f62-a0cd-509ba53e2cc1/Media/Images/library
DB_DATA_LOCATION=/srv/dev-disk-by-uuid-818c0386-a801-49c9-baa0-64c333614db5/immich/postgres
IMMICH_VERSION=release
DB_PASSWORD=xxxxx
DB_USERNAME=xxxxx
DB_DATABASE_NAME=xxxxx

rough spindle Oct 7, 2024, 11:13 PM

#

It looks like your database got corrupted

#

Any unexpected shutdowns / reboots / power loss?

#

What is the health of the immich_postgres container? Can you provide its logs also?

sullen kelp Oct 7, 2024, 11:15 PM

#

Not that I am aware of, before updating the container had an uptime of over a month and was working fine.

immich_postgres reports healthy, the log of the container is only 3 lines:

PostgreSQL Database directory appears to contain a database; Skipping initialization
2024-10-07 22:41:11.345 UTC [1] LOG:  redirecting log output to logging collector process
2024-10-07 22:41:11.345 UTC [1] HINT:  Future log output will appear in directory "log".``` I assume I should go check the log directory for more?

#

I just checked the log directory, this was the latest log. Similar to the server container, can't read a block

📎 message.txt

obsidian beacon Oct 8, 2024, 12:39 AM

#

Your docker-compose is also out of date for the immich-server service. Can you remove the command section?

sullen kelp Oct 8, 2024, 12:58 AM

#

obsidian beacon Your docker-compose is also out of date for the immich-server service. Can you r...

Ah, missed that, thanks!

sullen kelp Oct 8, 2024, 1:22 AM

#

I unfortunately, don't have a have a backup of the database. Is there anyway to have the photos that are at UPLOAD_LOCATION be re-added to new users? I get that albums and such probably have to be recreated, but at least having images for xyz user go back to xyz user would be helpful.

obsidian beacon Oct 8, 2024, 1:25 AM

#

sullen kelp I unfortunately, don't have a have a backup of the database. Is there anyway to ...

The easiest way is to use the CLI to upload the folder to the respective users. Or to mount them as external library.

#

However the bigger issue is how the database get corrupted

#

Since updating Immich shouldn’t cause that to happen.

#

Are you mounting your database location on a network drive by chance?

sullen kelp Oct 8, 2024, 1:28 AM

#

No, its on an ssd in the system

#Migration "AddThumbnailJobStatus1724080823160" failed

References

Checklist