#Postgress: could not open temporary statistics file "pg_stat_tmp/global.tmp": Input/output error

1 messages · Page 1 of 1 (latest)

covert echo
#

Postgres enters an I/O state trouble. This has happened before where the only solution is to remove power to the PI and then restart. Cannot stop or kill the docker container running it as demonstrated below

2025-11-30 05:01:15.066 GMT [1] LOG:  skipping missing configuration file "/var/lib/postgresql/data/postgresql.override.conf"
2025-11-30 05:01:15.067 GMT [1] LOG:  skipping missing configuration file "/var/lib/postgresql/data/postgresql.override.conf"
2025-11-30 05:01:20.495 UTC [1] LOG:  starting PostgreSQL 14.17 (Debian 14.17-1.pgdg120+1) on aarch64-unknown-linux-gnu, compiled by gcc (Debian 12.2.0-14) 12.2.0, 64-bit
2025-11-30 05:01:20.508 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
2025-11-30 05:01:20.508 UTC [1] LOG:  listening on IPv6 address "::", port 5432
2025-11-30 05:01:20.651 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2025-11-30 05:01:20.679 UTC [16] LOG:  database system was shut down at 2025-11-30 05:00:05 UTC
2025-11-30 05:01:21.033 UTC [1] LOG:  database system is ready to accept connections
2025-12-01 16:30:52.000 UTC [24] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": Input/output error
2025-12-01 16:30:52.643 UTC [24] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": Input/output error
2025-12-01 16:30:52.643 UTC [24] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": Input/output error
2025-12-01 16:30:53.287 UTC [24] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": Input/output error
2025-12-01 16:30:53.287 UTC [24] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": Input/output error
2025-12-01 16:30:53.933 UTC [24] LOG:  could not open temporary statistics file "pg_stat_tmp/global.tmp": Input/output error

❯ docker stop immich_postgres
Error response from daemon: cannot stop container: immich_postgres: tried to kill container, but did not receive an exit event
still cargoBOT
#

:wave: Hey @covert echo,

Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich immich.

References

#

Checklist

I have...

  1. :ballot_box_with_check: verified I'm on the latest release(note that mobile app releases may take some time).
  2. :ballot_box_with_check: read applicable release notes.
  3. :blue_square: reviewed the FAQs for known issues.
  4. :blue_square: reviewed Github for known issues.
  5. :ballot_box_with_check: tried accessing Immich via local ip (without a custom reverse proxy).
  6. :ballot_box_with_check: uploaded the relevant information (see below).
  7. :ballot_box_with_check: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable

(an item can be marked as "complete" by reacting with the appropriate number)

Information

In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:

  • Your docker-compose.yml and .env files.
  • Logs from all the containers and their status (see above).
  • All the troubleshooting steps you've tried so far.
  • Any recent changes you've made to Immich or your system.
  • Details about your system (both software/OS and hardware).
  • Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
  • The version of the Immich server, mobile app, and other relevant pieces.
  • Any other information that you think might be relevant.

Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)

If this ticket can be closed you can use the /close command, and re-open it later if needed.

stiff trail
#

Are you using an SD card?

covert echo
#

/Home-NAS is a ZFS file system mirrored SSD

#

compose.yml

  immich-ts:
    image: tailscale/tailscale:latest
    container_name: immich-ts
    hostname: immich
    environment:
      - TS_AUTHKEY=REDACTED
      - TS_STATE_DIR=/var/lib/tailscale
      - TS_SERVE_CONFIG=/config/immich.json
      - TS_USERSPACE=true

    volumes:
      - ./ts-config:/config
      - ./ts-state:/var/lib/tailscale
    restart: unless-stopped

  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:v2.3.1
    user: 1009:1009
    volumes:
      - /Home-NAS/generalstorage/immich-files/data/photos/upload:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
      - /Home-NAS/nextcloud-aio-data/USERNAME/files:/mnt/nextcloud:ro
        # - ./potential_mount:/mnt/nextcloud
    network_mode: service:immich-ts
    environment:
      DB_HOSTNAME: immich-database
      DB_PASSWORD: REDACTED
      DB_USERNAME: REDACTED
      DB_DATABASE_NAME: immich
      REDIS_HOSTNAME: immich-redis
    depends_on:
      - immich-redis
      - immich-database
    restart: unless-stopped
    healthcheck:
      disable: false

  immich-redis:
    container_name: immich_redis
    image: docker.io/valkey/valkey:8-bookworm@sha256:ff21bc0f8194dc9c105b769aeabf9585fea6a8ed649c0781caeac5cb3c247884
    healthcheck:
      test: redis-cli ping || exit 1
    restart: unless-stopped
    ports:
      - '6379:6379'

  immich-database:
    container_name: immich_postgres
    image: ghcr.io/immich-app/postgres:14-vectorchord0.3.0-pgvectors0.2.0@sha256:fa4f6e0971f454cd95fec5a9aaed2ed93d8f46725cc6bc61e0698e97dba96da1
    environment:
      POSTGRES_PASSWORD: REDACTED
      POSTGRES_USER: REDACTED
      POSTGRES_DB: immich
      POSTGRES_INITDB_ARGS: '--data-checksums'
    user: 1009:1009
    ports:
      - '5432:5432'
    volumes:
      - /Home-NAS/generalstorage/immich-files/database:/var/lib/postgresql/data
    restart: unless-stopped
stiff trail
#

Input output errors is bad news. Means low level corruption or block device issues typically

#

Not really anything Postgres can do to cause this typically

#

Is it physically attached to the pi or a network share?

covert echo
#

PCIE HAT 2x SSD

stiff trail
#

It’s probably dying if I had to guess. Pi is not designed for this work load

#

You can check ZFS logs scrub etc

covert echo
# 
NAME       SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
Home-NAS   928G   633G   295G        -         -    24%    68%  1.00x  SUSPENDED  -
#

What's strange is that this was set up within the past year. Is that enough time to kill it?

stiff trail
#

I would have expected it to be an sd card tbh. But in my experience pi’s can be fairly brittle

#

I doubt it’s the SSD itself. Could be the hat maybe

#

Can you do a scrub?

#

Check dmesg

covert echo
stiff trail
#

WARNING: Pool 'Home-NAS' has encountered an uncorrectable I/O failure and has been suspended.

#

Looks like both NVMEs are throwing errors, which means they are probably fine and the HAT is the most likely culprit

covert echo
#

Are they similarly lower powered machines? I like the idea of running something weak but strong enough to self-host a few docker containers

stiff trail
#

N100 is pretty weak

#

Idk why you want weak haha. Power usage I guess. N100 would be best

#

You can look in #1183317039163637790

covert echo
#

Yeah power usage. I run only a few docker containers on it (Immich, Actual, Nextcloud) so power wasn't my main concern. I like the idea of a weak fanless design

stiff trail
#

Yes, along those lines. That one has m.2 slots which is nice

covert echo
#

Is there a way to handle the IO better? Is there a way to force the IO to release so I can reboot? When I try rebooting via shutdown -h now it will not fully turn off becuase of the IO hang

stiff trail
#

There’s probably slightly cheaper ones you can find
Unlikely. Releasing the IO means data loss. ZFS wil not want to do this

#

It probably will time out eventually

#

But a need for hard reboot would not surprise me

covert echo
#

You eman it'll time out for the reboot or in general

stiff trail
#

The FS already won’t be clean

#

I mean the IO might timeout and allow reboot

covert echo
#

I just bought 2 new SSDs to increase the storage of my system. Given that the Beelink comes witha a 1 TB and I already have 1 TB, it probably makes sense to get something like the Beelink and use all the SSDs I have, right? (Partially rationalizing here)

stiff trail
#

Maybe? I’m not understanding exactly how you’ll be deploying the storage

covert echo
#

Sorry, I have 2 x 1 TB sticks already and would be adding 2 x 2 TB sticks. The beelink comes with a 1 TB (I think for the OS)
So in theory, you could have 2 vdevs: one for the 2 TB and one for the 1 TB. (with the 3rd 1 TB for the OS)

oblique sparrow
#

I would also suggest an arm board with a RK3588 such as the Orange Pi 5 but you'll likely get more I/O from the N100

covert echo
#

How would the Orange Pi use the mirrored ZFS 2280 drives? I only see a single slot for the ornage pi

oblique sparrow
#

One way is you split the PCI-E lanes. Though I'm not sure the hardware for that is easy to get.

#

Realistically, you'd probably want a different RK3588 board.