#HAOS reinstall / Proxmox consideration / Immich library backup via Restic

1 messages · Page 1 of 1 (latest)

sick gorge
onyx grail
#

what hardware are you running haos on

#

would do a normal clean installation of haos and put backup back.

#

if you have backup (and encryption key)

cyan forum
#

Most people don't even have a HDMI adapter for their pi. I figure it's the same idea here.

#

You also don't need to remove the internal disk.

onyx grail
#

and if sd card is empty, booting pi with network and keyboard will bring up online pi imager, where you directly can select haos for install. but there is the missing hdmi adapter...

gilded kiln
#

If you have a ubuntu box lying around, method 1 is easier than method 2.

#

if you don't, use method 2

barren eagle
#

Think it's selected as first method as it does not require removing the boot drive from the host. As this is the "Generic x86-64" install, not for a Pi or something.

sick gorge
sick gorge
#

Method 2 is using a thumb drive. No need to remove the main rive either IIRC

#

It's just a matter of flashing Ubuntu in the usb, or HAOS directly. That's my confusion

barren eagle
#

Where do you start Balena Etcher from then? You will need an operation system to run that.

gilded kiln
#

Just use the 2nd option, 99% of people will have an os that can run balena w/ a usb port.

cyan forum
gilded kiln
#

That's a good resource

#

Is that how you currently run HAOS?

#

I'm likely going to switch to HAOS at some point but I've been procrastinating

barren eagle
cyan forum
# gilded kiln Is that how you currently run HAOS?

I have a HAOS and Supervised/debian VM to be able to help people here and test things. Even destructive ones as I can easily restore a snapshot or backup. My production HA is a docker install in a debian VM. I have no plans to switch.

sick gorge
# cyan forum Also consider virtualizing HAOS: <https://gist.github.com/Impact123/c23c36eafe16...

This post was actually meaning to install HAOS again after trying to migrate to Proxmox. I still have proxmox installed, as my goal was to try and solve some issues with external drives. I thought it was a good excuse to change teams and finally use proxmox, but honestly as soon as I installed it, I saw I had to reserve resources for it and that hurts.

I have 16GB ram, 4 cores, and 256gb SSD. Having to sacrifice 2gb and some space for layer that really won't be doing anything bothers me a lot.

My main VM is going to be HA anyway, and only maybe I'd add extra containers. I'm not used to Linux anyway, so I was planning to run everything as addons still.

#

After that, I honestly saw no point of proxmox. Maybe I'm missing something

cyan forum
#

I don't really trust HAOS/supervisor much (see links in article) so having whole VM snapshots/backups alone is extremely helpful.

sick gorge
#

Or maybe the people running it just have way more resources and don't mind the extra layer

#

Can you guide me on the resource allocation issue? What's your case?

cyan forum
#

What issue exactly? 2G is not that much. Not sure what you mean with space allocation but I'm happy to elaborate if you do.

sick gorge
#

I'll read your guide first and get back to you when I'm in front. I'm out at.

2G stings when you have 16GB. I run Frigate and Immich and both use quite a bit, so I don't like giving up free resources (especially as I don't feel like proxmox is actually "doing" anything)

cyan forum
#

I'd run frigate and immich in CTs. There's basically no virtualization overhead and it's also easier to give it resources like a GPU. I never looked into what it uses the 2G for in detail. I know the processes but not why they use so much ram. I figure it could be lower but 🤷

sick gorge
cyan forum
#

With a CT you can technically allocate whatever you want as they only use what they need. Just like a normal process. With VMs they will use everything you give them over time as they use memory for caching and PVE counts that as used. At least for linux guests: https://www.linuxatemyram.com/
MY HAOS VM gets 1/2G min/max. 2-4G is a good value though. Check with the glances addon what it really needs.
Memory is really complicated and there's lots of variables and choices.
My frigate CT never uses more than 2G on average. I don't use immich. My recommendation is to give each 4G max, 2G min and increase if needed.

sick gorge
cyan forum
#

Like I said, for CTs the memory is not directly allocated but only a limit. For VMs it depends if balloning is enabled. If all try to use 8G then KSM and ballooning will try to handle it and if not then the OOM killer will kill something. Over allocating memory requires knowledge about your environment to work properly/reliably.

sick gorge
#

ooof man this is overwhelming. I get it's just a matter of learning, but I see so tiny benefit for me, that it feels daunting. Linux is completely new for me, and Proxmox forces you to manage so much for yourself.

#

there's too much I don't understand, that I feel I'm digging myself into a hole by myself, and I feel it's unnecesary

#

sure, I could spend a ton of time learning about it, but again, don't think I "need" it so far

#

I mean, getting into containers and docker commands is what broke my HAOS to start with 😂 I'm clearly out of my field here

cyan forum
#

HAOS makes for a bad general purpose system. The docker command is mostly for debugging there.

sick gorge
#

I basically tried moving my Immich library from an external HDD to a new internal SSD. -> With help from AI, I used rsync, but HAOS host doesn't allow it, so it created a temp container for it. The process started, but HAOS killed the container after 30m midway as it doesn't seem to support extra containers.

This corrupted the new SSD (it's not getting detected at all anymore, I'm returning it). And for some reason, I noticed my storage went from 55% to 92% full after that.

I went on a quest to find out why, with AI help ofc. I seems overlays were using 50GB. AI suggested I could prune, but I wasn't sure, so I didn't. I end up using script to find dangling overlays, and deleted them.

After this, HAOS doesn't boot anymore. 😄 Thanks AI. Lesson, be careful with SSH access, and trust AI even less.

normal cosmosBOT
#

ChatGPT and other "AI" systems do a fantastic job of generating well structured, convincing looking answers... that are either totally garbage or, if you're lucky, have subtle flaws. Don't use them to "help" others, and don't use them yourself.

barren eagle
#

installing anything yourself (aka, not via supervisor / HA / addon) will make the system unsupported. So if you want to do more, VM is the way to go. Memory is balloning s real overhead is not that bad

sick gorge
# normal cosmos

I know I know. But sometimes it's the only way. No-one will hand hold you long enough to do what you need sometimes. 🙁

cyan forum
#

I always have to check if a command line option an AI gives me really exists. Last time ChatGPT gave me an imaginary option when I asked for a more succinct way to write my one-liner. It said something like

You could use cmd --flaghere blablabla (if it had that option)
I didn't read that far though and was momentarily confused. I like to use it to format command outputs of people who don't use code blocks and such. Unless you can tell if it gives you crap I wouldn't recommend it to learn something. It can also hallucinate service names if you ask it to restart all services of not well known program x for example. I have more boring stories but this has already gotten very off-topic.

sick gorge
#

yeah it definitely does that, I've had that plenty of times too

sick gorge
#

oh wait, getting back on topic, the actual method 2 is NOT from a usb flash drive

#

now I understand what you all meant

#

the balena etcher method is actually installing HAOS in the actual drive

#

weird, I recally 3y ago when I set up I think I used the etcher to flash a boot image, but then I installed HAOS in the insternal drive

#

that's what I was expecting with this option 2

#

maybe I don't remember well

cyan forum
#

The image is basically just a representation of a disk with a few partitions you're writing onto another disk. Flashing is installig in this case. Method one uses a live ubuntu and its disks utility to flash, method two uses etcher installed on your PC.

sick gorge
# 
If you are getting an Error unmounting filesystem error message, stating that the target is busy:
Most likely, you are running Ubuntu on your internal disk. Instead, you need to run it on your stick.

    Go back to step 3 and during start up, make sure you select Try Ubuntu (and NOT Install Ubuntu).

When trying this, the BIOS boot option basically says "Try or Install Ubuntu", which is the one I used previously.

#

I'm just trying again

#

yeah it just worked, weird

sick gorge
#

tbh I think this reinstall using the backup was a good exercise and cleanup. I went from 230GB used to 38GB. Wtf

#

there was some serious shit in there

cyan forum
sick gorge
#

I did investigate, but the result was finding the overlays xDDD

sick gorge
# cyan forum <https://pve.proxmox.com/wiki/Dynamic_Memory_Management> <https://gist.github.co...

Hey. I'm considering going Proxmox once again. I've got stuck for trying to build a backup strategy for Immich, and it seems the proper tools to do it are restic or borg, both of which don't have existing addons, so I don't have a proper alternative.

I'm planning to follow your guide ofc. But... would you be so kind as to be there for some handholding? I'm sure I'll need it at some point for some silly questions. I'd feel way safer having someone experience instead of relying in chatGPT for this. Especially when it means shutting HAOS down in the meantime until restored in VM.

I'd open a new thread ofc.

cyan forum
#

It's not my guide. PVE has a integrated whole guest backup feature. There's also the proxcord discord server I'm active in if it's PVE specific.

sick gorge
cyan forum
#

Feel free to tag me if you need help.

sick gorge
# cyan forum Feel free to tag me if you need help.

Man, I asked chatGPT to create me a "tutorial" of sorts using the guides etc. Not to follow blindly, but just to see where I was getting into and have an overview before shutting off HAOS.

And I SHAT myself.

I'll leave it here if you're curious to see how off it went. I'm sure it will all look easy for you...

In any case, this is way overkill/too complex for what I need (just run restic for backups from Immich). I know I'm going to get stuck 1000s of times in those tiny commands that I don't even know what they do.

I'm pretty much back at finding an alternative for my Immich backups. I even see easier to create a restic addon myself (via AI) than this Proxmox odyssey. 🙁

cyan forum
#

Proxmox VE is not really something you use as alternative to restic. Restic is fine to back up files.

sick gorge
#

How can I run restic in HAOS then? There's no addon (there are a few but old an unmaintained)

cyan forum
#

You can probably install it in the SSH addon but automating it might be awkward.

sick gorge
#

is it not a container!!??? wtf

#

I thought it was!

cyan forum
#

It is.

sick gorge
#

hm? HAOS doesn't allow creating custom containers AFAIK, otherwise I'd use portainer to do it as it's easier for me

cyan forum
#

No I mean you can use apk add restic or whatever. I kinda lost the plot about exactly what you want to do. It's been a few days and lots of topic switches.

sick gorge
# cyan forum No I mean you can use `apk add restic` or whatever. I kinda lost the plot about ...

yeah, sorry, just keeping it here as there's the full context.
My single goal is to create backups from my Immich Library. This is mounted in /media/. I tried doing it with rclone, but the Immich mods told me it's not a proper backup tool, and that restic or borg are the right thing to use.

I then tried looking for addons of those, but there aren't. So I felt stuck, and thought my only way was setting up PVE so I'd be able to set up restic as a container and backup Immich library that way. But as mentioned before, it's overly complex just to solve this.

#

restic seems the perfect tool, as it's compatible with Google Shared Drives, which is my backup destination. It's just a matter of how to use it within HAOS

cyan forum
#

I keep telling people that HAOS makes for a poor NAS or general purpose OS. Where do you want to back up to?

sick gorge
#

Google Shared Drive

#

I know, but I need little more than this. And Proxmox feels overkills for that. I don't really want a NAS. Just HAOS with a few sparks

cyan forum
#

Use the Advanced SSH addon and run apk add restic. Tada, you have it.

sick gorge
#

will it persist?

#

and there's also the thing about automating it somehow as you mentioned

cyan forum
#

To automate it you can call a script that runs a SSH command to the addon.

sick gorge
#

man, this just saved my life. And as usual, it's WAY easier than it seemed

cyan forum
#

Just make sure to store the config in a path that is preserved such as /share.

sick gorge
#

I'm confused and not handy with these terms, sorry

cyan forum
#

Every addon is a docker container.

sick gorge
#

but restic is not an addon

#

just a binary

cyan forum
#

Yep.

sick gorge
#

so? Which config did you mean? The SSH addon config maybe?

cyan forum
#

The restic config.

#

If you put it in /root it will not necessarily persist, for example.

sick gorge
#

I just don't get how to use this docker inspect homeassistant | jq .[].Mounts

#

how is that related to restic (that is not an addon)?

cyan forum
#

It is related to the SSH addon itself. It is to show you where you can put the restic config so it will not perish when the addon is rebuild.

sick gorge
#

ahh, I got it now

cyan forum
#

You need to read the other info to understand the commands. Hence the depends on info above note.

sick gorge
#

I found this, but this is just the addon config. That persists as usual. That isn't what you meant, isn't it?

root@a0d7b954-ssh:/$ ls /data/
options.json              ssh_host_ed25519_key      ssh_host_ed25519_key.pub  ssh_host_rsa_key          ssh_host_rsa_key.pub
root@a0d7b954-ssh:/$ cat /data/options.json
{
  "ssh": {
    "username": "root",
    "password": "",
    "authorized_keys": [
      "redacted"
    ],
    "sftp": true,
    "compatibility_mode": false,
    "allow_agent_forwarding": false,
    "allow_remote_port_forwarding": false,
    "allow_tcp_forwarding": false
  },
  "zsh": true,
  "share_sessions": false,
  "packages": [
    "restic"
  ],
  "init_commands": []
}root@a0d7b954-ssh:/$ ^C
root@a0d7b954-ssh:/$
cyan forum
#

No.

sick gorge
#

well, I'll get down to testing restic first and then maybe I see a new file with its config or something. And then I'll get back to see how to save it

peak adder
# sick gorge Man, I asked chatGPT to create me a "tutorial" of sorts using the guides etc. No...

The ChatGPT guide could have been simpler:

1). Install Proxmox - just do the basic QuickStart install
2). Add the 2TB drive
3). Visit https://community-scripts.github.io/ProxmoxVE/ and use the scripts to install software (use LXC's over VM's where possible).

The script for Frigate should enable GPU Passthrough.

Proxmox VE Helper-Scripts
Proxmox VE Helper-Scripts

The official website for the Proxmox VE Helper-Scripts (Community) Repository. Featuring over 300+ scripts to help you manage your Proxmox VE environment.

sick gorge
# peak adder The ChatGPT guide could have been simpler: 1). Install Proxmox - just do the ba...

that's just oversimplifying it imo. The Frigate docs themselves recommend against using LXC as method, so going that way itself is asking for trouble.

I agree the guide could be simplified though. I was more interested in the sources really.

Installation | Frigate

Frigate is a Docker container that can be run on any Docker host including as a Home Assistant Add-on. Note that the Home Assistant Add-on is not the same thing as the integration. The integration is required to integrate Frigate into Home Assistant, whether you are running Frigate as a standalone Docker container or as a Home Assistant Add-on.

cyan forum
#

Works fine in a CT. The maintainer's just don't use PVE from what I can tell.

sick gorge
#

the point is that you need to know what you're doing if "going against the recommendation", and I don't. 😂

sick gorge
# cyan forum It is related to the SSH addon itself. It is to show you where you can put the r...

do you know why this might not be working?

OK: 26347 distinct packages available
(1/3) Installing restic (0.18.0-r3)
(2/3) Installing restic-bash-completion (0.18.0-r3)
(3/3) Installing restic-zsh-completion (0.18.0-r3)
Executing busybox-1.37.0-r18.trigger
OK: 305 MiB in 248 packages
(1/3) Installing rclone (1.69.3-r1)
(2/3) Installing rclone-bash-completion (1.69.3-r1)
(3/3) Installing rclone-zsh-completion (1.69.3-r1)
Executing busybox-1.37.0-r18.trigger
OK: 392 MiB in 251 packages

The log doesn't say it ran the init command, and when checking it actually didn't.

image.png
#

I'm doing rclone first and then I'll do restic the same way

#

if I run the command manually it works

#

Found why. That env is set in a one-shot shell, and is lost when I open my shell. So init_commands is not useful for this.

cyan forum
#

You can echo that into the bashrc or .profile though.

#

Or just use it in-line with your command.

#

ENVVAR=XXX commandhere...

sick gorge
#

as far as I've seen, restic has no file config like rclone though. When you said so, what did you mean?

cyan forum
#

What's ls -lah say?

#

You know what, I mixed them up.

#

It might make sense to put its cache directory in a persisted path though.

sick gorge
sick gorge
cyan forum
sick gorge
#

oh, got it. That could be useful, yeah. Will look into it later. Hopefully I'll be able to move it and tell restic to use the permanent path

cyan forum
#

I'd recommend you write a small script and execute that. There you can also set the environment variables for all of that.

sick gorge
#

I'll be calling it with a shell script later for the automation, yeah. But atm I'm just testing the setup manually.

#

looking very promising though. And such a relief from yesterday, this is SO much easier than the Proxmox ordeal.

#

HAOS reinstall / Proxmox consideration / Immich library backup via Restic

sick gorge
#

script example:

#!/bin/bash
export RESTIC_CACHE_DIR=/share/restic/cache
export RESTIC_REPOSITORY=rclone:BlowUpTD.Backups.Immich:
export RESTIC_PASSWORD_FILE=/share/restic/repo-key.txt

restic \
  -o rclone.args="--config=/share/rclone/rclone.conf serve restic --stdio" \
  backup /media/STORAGE/immich/backups/ \
  --exclude-file=excludes.txt
cyan forum
# sick gorge am I correct in saying that I cannot just run these 2 scripts? restic and rclone...

You know what, we don't even need the SSH addon for this.
shell_command runs the command in the HA container, not the SSH addons so the command doesn't exist there but you could add something like this at the top of your script

#!/usr/bin/env bash

(command -v restic || apk add --no-cache restic) &>/dev/null

export RESTIC_CACHE_DIR=/share/restic/cache
export RESTIC_REPOSITORY=rclone:BlowUpTD.Backups.Immich:
export RESTIC_PASSWORD_FILE=/share/restic/repo-key.txt

restic \
  -o rclone.args="--config=/share/rclone/rclone.conf serve restic --stdio" \
  backup /media/STORAGE/immich/backups/ \
  --exclude-file=excludes.txt

You could also just run this with another automation whenever HA starts.
Is there a good reason for splitting this into two scripts?
Your exclude file should probably be in /share/restic too.

sick gorge
#

yeah I figured that too. But is there any benefit of running it into HA container?
Do the packages persist there? I thought I couldn't install any tbh.

I have it working fine with the SSH addon, so unless there's any benefit... I'll stick with it.

Splitting the scripts is to be able to call them independently. Backup will be daily, but retention/pruning weekly.
Yeah I fixed the excludes already.
I'm finishing touches with logging and notifications now.

cyan forum
#

The benefit is that you don't have to SSH into the SSH container. Not sure why I didn't think of it. I helped people with this before.

sick gorge
# 
#!/bin/bash

LOGFILE="/config/immich_backup/immich_backup.log"
exec >>"$LOGFILE" 2>&1 # All logs after this will be appended to the logfile

# Log heading
echo ""
echo "---------------------------------------------------------"
echo "=== Retention/prune started at $(date '+%Y-%m-%d %H:%M:%S') ==="

export RESTIC_CACHE_DIR=/share/restic/cache
export RESTIC_PASSWORD_FILE=/share/restic/repo-key.txt
export RESTIC_REPOSITORY=rclone:BlowUpTD.Backups.Immich:

# Restic command captured into Summary for logging
SUMMARY=$(
  restic \
    -o rclone.args="--config=/share/rclone/rclone.conf serve restic --stdio" \
    forget \
    --keep-daily 7 \
    --keep-weekly 4 \
    --keep-monthly 12 \
    --prune \
    2>&1
)
RET_CODE=$?

# Write full summary to log
echo "$SUMMARY" >> "$LOGFILE"

# HA REST API info
HA_URL="http://homeassistant.local:8123"
HA_TOKEN="eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJkNjg5OWExN2U5NmQ0N2VkYWVkY2UzZjQ2MjgxOWNmZSIsImlhdCI6MTc1NjY2MDIwOSwiZXhwIjoyMDcyMDIwMjA5fQ.IPD0K-3n6Yii7wQYA6MLGG-v2O453Od0leLp7cGUfAI"

# Notify HA
notify_ha() {
    local message="$1"
    curl -s -X POST \
        -H "Authorization: Bearer $HA_TOKEN" \
        -H "Content-Type: application/json" \
        -d "{\"message\":\"$message\",\"title\":\"Immich Backup\"}" \
        $HA_URL/api/services/persistent_notification/create
}

if [ $RET_CODE -eq 0 ]; then
    notify_ha "Immich retention/prune completed successfully at $(date '+%Y-%m-%d %H:%M:%S')\n$SUMMARY"
    echo "=== Retention completed successfully at $(date '+%Y-%m-%d %H:%M:%S') ===" >> "$LOGFILE"
else
    notify_ha "Immich retention/prune FAILED at $(date '+%Y-%m-%d %H:%M:%S')\n$SUMMARY"
    echo "=== Retention FAILED at $(date '+%Y-%m-%d %H:%M:%S') ===" >> "$LOGFILE"
fi
cyan forum
#

Use bash right after the three backticks on the same line to color it just like mine above.

sick gorge
cyan forum
#

You can't really make it persist for core but the way I shared above is trivial. Just one line.

sick gorge
#

ah you mean it'd install them every time the script runs

cyan forum
#

When it doesn't already exists. Yep.

sick gorge
#

I get it. But again, is there any other downside from running in the addon? Tbh I even feel it safer than messing with HA

sick gorge
# cyan forum

was this showing something? Or just to show the ticks bash?

cyan forum
#

Not that I can think of. It's just a dependency you don't need.

sick gorge
#

I just did a test backup locally to test restic and it's amazing. Much quicker than rclone scanning files (250k in my backup), and on top of that it actually backs up 18k files only instead of the the actual 250k, which is a lifesaver for me since Google Team Drives have a hard cap of 500k files, and I was getting dangerously close.

cyan forum
#

I don't really get how it's more hassle to do it without the addon now. Your files are in /share and the command is simpler.
Feels more like sunk-cost fallacy.

sick gorge
#

hmmm

cyan forum
#

You can likely also use a better/easier way to notify but that is not my expertise.

sick gorge
#

confirm me one thing before attempting the change.

I have this other script I did previously for the Immich db dump. Will this one work just the same in the HA container?

As this one has a completely different command.

#!/bin/bash
BACKUP_DIR="/media/STORAGE/immich/db_backup"
TIMESTAMP=$(date +"%Y%m%d_%H%M%S")

# Make sure the directory exists
mkdir -p "$BACKUP_DIR"

# Dump all Postgres databases from Immich container
docker exec -d addon_db21ed7f_postgres_latest sh -c "pg_dumpall --clean --if-exists --username=postgres | gzip > '$BACKUP_DIR/db_dump_$TIMESTAMP.sql.gz'"
cyan forum
#

No. docker will not work inside HA.

sick gorge
sick gorge
cyan forum
sick gorge
#

ok. I'll see if I change the other 2 anyway.

cyan forum
#

I assume it backs up all of the data and it's too much?

sick gorge
#

hmm. I'm not an expert here. That is the command recommended by Immich docs. The db dump needs to be take at the same time the library is backed up, with the server (Immich addon) stopped, so they are in sync. Otherwise, on restore, you get errors of missing files/records and it's a mess.

Doing it with HA would mean triggering a single addon backup for that. Not sure if that'd work tbh. And not sure I can direct that I want that dump in /media/STORAGE/immich/db_backup, instead of the default HA backup folder.

sick gorge
# cyan forum <https://www.home-assistant.io/integrations/shell_command/#automation-example> Y...

It won't work in this case (unless I stand corrected). For all these shell commands, I'm forced to run them with nohup, because HA has a hard timeout at 60s. They all take longer, so I need them running in the bg, and then I get no logs about them. That's why I can't use the response_variable, and why I set up a log file. 🙁

    actions:
      - action: shell_command.get_file_contents
        data:
          filename: "todo.txt"
        response_variable: todo_response
cyan forum
#

Just think about restoring and if that would be painful. I don't use immich so this was just something that stuck out to me.

sick gorge
#

nah I already did a db restore a few days ago. It's fine.
It's the same as we were saying with the Frigate LXC setup. It can be done, but I'd rather stick to the recommended path officially. That way I can ask for support if needed (probable). If I go my own way and I mess up, I'm f'ed.

#

are you positive (command -v restic || apk add --no-cache restic) &>/dev/null this won't flag HA as unsafe, right? As if I installed a custom container? Already had that issue in the past

#

hmm I was thinking, having it in the SSH addon allows me to jump in and check things manually with restic and rclone. I wouldn't be able to do that in the HA container, right?

cyan forum
cyan forum
sick gorge
#

oh ok. No prob then.

sick gorge