#Redis used all my disk space, how to handle it?

1 messages · Page 1 of 1 (latest)

late root
#

Hello all, I was running a smart search job, when I noticed my server was stuck for a while with no CPU usage. I tried restarting and such, only to check some logs and find that Redis failed, because... it has used up my entire boot drive (290GB as it's only needed for boot).

Either way, I have Immich's database on separate (larger) drives, so it's only the boot drive affected. I just mainly want to know how to handle the full boot drive?
Should the Redis data be deleted, should it be moved elsewhere? Can Redis be pointed to store data on a separate drive?

Sorry if this may seem a little dumb lol. Thanks for any help :)

kind obsidianBOT
#

:wave: Hey @late root,

Thanks for reaching out to us. Please carefully read this message and follow the recommended actions. This will help us be more effective in our support effort and leave more time for building Immich immich.

References

#

Checklist

I have...

  1. :blue_square: verified I'm on the latest release(note that mobile app releases may take some time).
  2. :blue_square: read applicable release notes.
  3. :blue_square: reviewed the FAQs for known issues.
  4. :blue_square: reviewed Github for known issues.
  5. :blue_square: tried accessing Immich via local ip (without a custom reverse proxy).
  6. :blue_square: uploaded the relevant information (see below).
  7. :blue_square: tried an incognito window, disabled extensions, cleared mobile app cache, logged out and back in, different browsers, etc. as applicable

(an item can be marked as "complete" by reacting with the appropriate number)

Information

In order to be able to effectively help you, we need you to provide clear information to show what the problem is. The exact details needed vary per case, but here is a list of things to consider:

  • Your docker-compose.yml and .env files.
  • Logs from all the containers and their status (see above).
  • All the troubleshooting steps you've tried so far.
  • Any recent changes you've made to Immich or your system.
  • Details about your system (both software/OS and hardware).
  • Details about your storage (filesystems, type of disks, output of commands like fdisk -l and df -h).
  • The version of the Immich server, mobile app, and other relevant pieces.
  • Any other information that you think might be relevant.

Please paste files and logs with proper code formatting, and especially avoid blurry screenshots.
Without the right information we can't work out what the problem is. Help us help you ;)

If this ticket can be closed you can use the /close command, and re-open it later if needed.

unique rampart
#

I doubt redis is talking up this much space

#

It's much more likely you have a bunch of old unused docker images

gaunt drift
#

What's the output of docker system df ?

late root
# unique rampart It's much more likely you have a bunch of old unused docker images

I've removed some unnecessary files, but i can't find what's taking up the space. I'll troubleshoot it later, as it may be something else on the server. I though it was Redis because the drive only became full after it was running smart search, nothing else was done between then. I've managed to recover 20GB for now, so theres that.

late root
# gaunt drift What's the output of `docker system df` ?

TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 6 6 4.013GB 74.78MB (1%)
Containers 6 5 24.46kB 0B (0%)
Local Volumes 6 3 4.365GB 30B (0%)
Build Cache 0 0 0B 0B

Yeah not much
But like I said in my other message, I'll look into this a bit later, as it may be something else.

#

For now though, can you point Redis at a different location rather than the boot drive? Just to avoid any future issues, if not it's fine

ashen estuary
#

Sure but there’s no evidence that is the actual issue

#

You should start to research what is actually taking up the space then report back if it’s related to immich in any way

late root
#

I'm currently doing so

gaunt drift
late root
#

No, just found the issue

gaunt drift
#

Unless you changed redis away from the default and set it to use a mounted folder, which I really doubt, any redis storage would be included there

late root
#

There is another folder there which was taking majority space. Just hold one one sec, I'm checking things

#

Okay I removed the folder, it was taking about 56GB, not sure how the disk got full. I can't remember where it was at beforehand, but I'm sure it wasn't that full. Either way, removed the folder and Smart Search is running again with no errors (so far). I'll monitor this closely and update if needed.

Sorry for any hassles, thanks for the help

(OUTDATED, see below update)

late root
#

It has once again done this.

image.png
#

I've been tearing my hair our the entire morning trying to resolve this

#

So i re-ran smart search and it was going fine until about 6am this morning (3 hours ago from now) when i noticed it failed again, for the same reason. Checked and sure enough, the entire drive was filled.

So here I am again. Now the drive had about ~70GB free when i started yesterday again. (Nothing was added, as I was sleeping. Unless i somehow added 70gb while sleepwalking)I am not sure how much data Redis uses, but surely it can't use this much?

#

Still trying to figure this out, but will post further info soon ,as I'm still going through everything. Please let me know, if I should provide any specific info here

late root
#

My current issue is finding what exactly is taking up all the space, as the biggest things on the root drive are as follows:

13G /home
12G /var
7.6G /usr

Yeah, so either I'm an idiot whos missed something very obvious, or... I dont even know anymore

gaunt drift
#

What are the outputs of df -h, lsblk, and mount?

late root
#

Filesystem Size Used Avail Use% Mounted on
udev 7.4G 0 7.4G 0% /dev
tmpfs 1.5G 1.5M 1.5G 1% /run
/dev/sdb1 292G 288G 0 100% /
tmpfs 7.5G 0 7.5G 0% /dev/shm
tmpfs 5.0M 12K 5.0M 1% /run/lock
/dev/sda 880G 384G 451G 46% /mnt/database
/dev/sdc1 458G 199G 236G 46% /mnt/media
tmpfs 1.5G 84K 1.5G 1% /run/user/1000

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 894.3G 0 disk /mnt/database
sdb 8:16 0 298.1G 0 disk
├─sdb1 8:17 0 297.1G 0 part /
├─sdb2 8:18 0 1K 0 part
└─sdb5 8:21 0 976M 0 part [SWAP]
sdc 8:32 0 465.8G 0 disk
└─sdc1 8:33 0 465.8G 0 part /mnt/media
sdd 8:48 0 953.9G 0 disk
└─sdd1 8:49 0 953.9G 0 part /media/root/84ee8b8d-42f1-4641-bebc-8af52ef7899c

(will send in a moment)

and as said above with du on everything

838G /mnt
384G /media
13G /home
12G /var
7.6G /usr
1.6G /root
686M /opt
153M /boot
12M /etc
1.6M /run
76K /tmp
16K /lost+found
4.0K /srv

#

and docker df:

TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 6 2 4.013GB 3.684GB (91%)
Containers 2 1 0B 0B
Local Volumes 8 1 4.365GB 4.365GB (99%)
Build Cache 0 0 0B 0B

#

Sorry if I'm somehow missing something here

gaunt drift
#

What does du -hd1 /mnt say?

late root
#

255G /mnt/backup
199G /mnt/media
384G /mnt/database
838G /mnt

Each one mounted is a separate drive

gaunt drift
#

/mnt/backup doesn't have a disk mounted on it

#

So anything written there is going into your root fs

#

Which is why it's filling up

late root
#

the what

#

One sec, let me check

#

Oh my god