Backup Settings | authentik | Page 1

past ravine Jul 14, 2024, 5:57 PM

#

well first off what are the errors

#

if its caused by your data you shouldn't back it up

river sleet Jul 14, 2024, 11:26 PM

#

the server and worker containers status goes to unhealthy after several hours. The warning I can see is:

#

warning error=authentik starting event=failed to proxy to backend logger=authentik.router

#

It was working fine for months and recently started happening...nothing has changed except for recent authentik upgrades.

river sleet Jul 15, 2024, 1:47 AM

#

digging further it looks like the postgres container is most likely the culprit. It is throwing: FATAL: sorry, too many clients already.

past ravine Jul 15, 2024, 1:47 AM

#

are you on pg16

river sleet Jul 15, 2024, 1:47 AM

#

Has anyone ran into this issue?

past ravine Jul 15, 2024, 1:49 AM

#

past ravine are you on pg16

and whats ur current authentik version / does authentik have it's own container or you're using a common db

river sleet Jul 15, 2024, 1:49 AM

#

probably not...this is what i have in my script

#

docker.io/library/postgres:12-alpine

#

authentik on 2024.6.1

#

yes authentik is on it's own container

past ravine Jul 15, 2024, 1:50 AM

#

you had to upgrade your postgres when passing to authentik 2024.6.x

#

https://docs.goauthentik.io/docs/troubleshooting/postgres/upgrade_docker

Upgrade PostgreSQL on Docker Compose | authentik

Dump your database

river sleet Jul 15, 2024, 1:52 AM

#

🤦‍♂️ I didn't realize I needed to upgrade the database

#

thanks for pointing that out

past ravine Jul 15, 2024, 1:52 AM

#

no problem

#

ping me if it still happens after the upgrade ill try to help you out more

river sleet Jul 15, 2024, 8:00 PM

#

I upgraded the ps to 16 and that resolved that issue but authentik still was hanging after an hour so I decided to start over and reconfigure everything. Still not working

past ravine Jul 15, 2024, 8:01 PM

#

can you share your logs

river sleet Jul 15, 2024, 8:01 PM

#

I noticed this in the log.
/ak-root/venv/lib/python3.12/site-packages/opencontainers/distribution/reggie/defaults.py:17: SyntaxWarning: invalid escape sequence '('

"http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*(),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+"

#

any ideas?

past ravine Jul 15, 2024, 8:01 PM

#

thats normal-ish

river sleet Jul 15, 2024, 8:01 PM

#

which container logs?

past ravine Jul 15, 2024, 8:01 PM

#

past ravine thats normal-ish

quoting jens:

Yeah it’s due to a dependency that hasn’t been updated

past ravine Jul 15, 2024, 8:02 PM

#

river sleet which container logs?

well wdym by hanging

river sleet Jul 15, 2024, 8:02 PM

#

the server container goes to unhealthy status and I am not able to get to my authentik website or any other sites that I set up

past ravine Jul 15, 2024, 8:03 PM

#

docker compose up --build --force-recreate

#

and then send logs of everything

#

and can you also send your docker-compose.yml

river sleet Jul 15, 2024, 8:09 PM

#

it's taking a minute...can't stop the containers...having to reboot vm.

#

📎 authentik_script.txt

#

📎 authentik-worker-1_logs.txt

#

📎 authentik-server-1_logs1.txt

#

📎 authentik-postgresql-1_logs.txt

past ravine Jul 15, 2024, 8:15 PM

#

let it run a minute to get the logs and not just startup things

#

and can you try copying it in a way that doesn't break formatting / encoding

river sleet Jul 15, 2024, 8:17 PM

#

it usually takes an hour before it becomes unresponsive

river sleet Jul 15, 2024, 10:17 PM

#

@past ravine The server container just went to unhealthy status but don't see anything in the logs

📎 authentik-postgresql-1_logs.txt

#

📎 authentik-server-1_logs1.txt

#

📎 authentik-worker-1_logs1.txt

past ravine Jul 16, 2024, 12:59 AM

#

hmm i don't seem to see any errors either. could you share your docker information and virtual machine information

#

i saw stack.env in your compose file. are you deploying using portainer? if so, can you try deploying from the command line directly

#

@river sleet

river sleet Jul 16, 2024, 1:18 AM

#

Yea I set it up in portainer. I'll remove it and try from the command line.

#

Just weird that it was working fine for more than 6 months

river sleet Jul 16, 2024, 4:16 AM

#

@past ravine I just tried the command line and it still goes unresponsive after an hour or so.

#

I noticed this warning
{"domain_url": null, "event": "Loaded MMDB database", "file": "/geoip/GeoLite2-City.mmdb", "last_write": 1720729303.0, "level": "info", "logger": "authentik.events.context_processors.mmdb", "pid": 15, "schema_name": "public", "timestamp": "2024-07-16T02:14:39.100134"}

warning error=authentik starting event=failed to proxy to backend logger=authentik.router timestamp=2024-07-16T02:14:40Z

#

any idea what the issue could be

past ravine Jul 16, 2024, 4:22 AM

#

by the looks of it it's just authentik starting

past ravine Jul 16, 2024, 4:23 AM

#

past ravine hmm i don't seem to see any errors either. could you share your docker informati...

but can you

river sleet Jul 16, 2024, 4:30 AM

#

I'm running proxmox 8.2.4 using an Ubuntu VM

#

4 processor and 16gb of RAM

past ravine Jul 16, 2024, 4:31 AM

#

uhhh i'm really not sure now

#

mabye someone else would have an idea?

river sleet Jul 16, 2024, 4:31 AM

#

ok thanks for trying

past ravine Jul 16, 2024, 4:32 AM

#

can you try deleting all data and downgrading to 2024.4.x and retrying?

river sleet Jul 16, 2024, 4:32 AM

#

yes ...I was thinking of reverting

past ravine Jul 16, 2024, 4:32 AM

#

it might be an issue with proxmox and the latest release

#

but make sure ur on latest docker stable

river sleet Jul 16, 2024, 4:32 AM

#

with 2024.4.x which pg version should I be using

past ravine Jul 16, 2024, 4:32 AM

#

might as well stay with 16

#

i upgraded after the pr was merged and didn't have any issues

river sleet Jul 16, 2024, 4:33 AM

#

Server: Docker Engine - Community
Engine:
Version: 27.0.3
API version: 1.46 (minimum version 1.24)
Go version: go1.21.11
Git commit: 662f78c
Built: Sat Jun 29 00:02:33 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.7.18
GitCommit: ae71819c4f5e67bb4d5ae76a6b735f29cc25774e
runc:
Version: 1.7.18
GitCommit: v1.1.13-0-g58aa920
docker-init:
Version: 0.19.0
GitCommit: de40ad0

#

What was the latest 2024.4.x version?

past ravine Jul 16, 2024, 4:34 AM

#

.2 i think

#

nvm it's .3

river sleet Jul 16, 2024, 4:35 AM

#

ok thanks...I think that will solve my problem since for longest time had no issues

#

BTW, do you use proxmox too?

#

or what is your setup

past ravine Jul 16, 2024, 4:36 AM

#

but are you able to get authentik working with that version

river sleet Jul 16, 2024, 4:36 AM

#

yes I was using authentik since 2024.2.x

past ravine Jul 16, 2024, 4:37 AM

#

river sleet or what is your setup

rn it's cloud vms

#

moved them from my house to there for security, performance, and reliability

#

3 x 24gb ram nothing fancy really

river sleet Jul 16, 2024, 4:39 AM

#

I see.

past ravine Jul 16, 2024, 4:39 AM

#

i'm planning on moving them to a baremetal ec2 one day

#

and installing pve again

#

@river sleet so is authentik working fine now?

#

@final carbon do you know of any issues that could cause this on 2024.6? or maybe i missed something

river sleet Jul 16, 2024, 4:43 AM

#

I don't know yet...I just fired up my pbs and going to roll back to an instanct with my portainer config

past ravine Jul 16, 2024, 4:43 AM

#

i'm curious why are you managing it with portainer instead of the command line

river sleet Jul 16, 2024, 4:43 AM

#

and then update that script...I prefer portainer because it's easier

past ravine Jul 16, 2024, 4:43 AM

#

last time i played with portainer i deleted my gitlab install

river sleet Jul 16, 2024, 4:43 AM

#

I don't have to ssh in...got a web gui

past ravine Jul 16, 2024, 4:44 AM

#

makes sense

#

i just do ci/cd/devops for updates so i never ssh either

river sleet Jul 16, 2024, 4:44 AM

#

yea I'm lazy LOL

past ravine Jul 16, 2024, 4:45 AM

#

i can't even log in lol

#

all my stuff is behind vpn so i only have authentik for a few things but i'm planning on adding it to the remaining things and maybe depend less on vpn

river sleet Jul 16, 2024, 4:47 AM

#

I'm like you...I only have a few things out in internet...the rest I wireguard to access.

past ravine Jul 16, 2024, 4:47 AM

#

i never got wireguard to work

#

i only have 2 public things: my broken and incomplete and weird website and my image host

river sleet Jul 16, 2024, 4:47 AM

#

I have wireguard on an LXC...works great

past ravine Jul 16, 2024, 4:48 AM

#

past ravine i never got wireguard to work

like i allowed the ports thru ip tables and ufw but i still couldn't get internet connection

#

well dns resolution actually

river sleet Jul 16, 2024, 4:49 AM

#

did you try tailscale and headscale? that works great too

past ravine Jul 16, 2024, 4:49 AM

#

unless it's cause my adguard runs on a container in a docker network and i expose :53 but if i expose it on all interfaces it causes massive drops in internet conn

river sleet Jul 16, 2024, 4:49 AM

#

but it was overkill for what I needed

past ravine Jul 16, 2024, 4:49 AM

#

yea i use tailscale

#

gonna add 3 more ips to it when i protect my intranet better

#

reminder to self tdk-1038 which has been unresolved for over a week

river sleet Jul 16, 2024, 4:50 AM

#

you should setup headscale if you have not

past ravine Jul 16, 2024, 4:51 AM

#

tbh i don't really see the need to self host it

river sleet Jul 16, 2024, 4:52 AM

#

I just prefer not having things routed outside of my self host if not needed

past ravine Jul 16, 2024, 4:52 AM

#

you still communicate with tailscale

#

when you set the control plane server in the ios app for example

#

changing the url looks like a pain in the a in windows

river sleet Jul 16, 2024, 4:53 AM

#

I don't use tailscale anymore but I remember that you have to go thru the tailscale server for routing

past ravine Jul 16, 2024, 4:53 AM

#

yea

#

i'm not too much of a networking nerd so i don't want to make it too hard on myself either

#

but out of curiosity what do you self host

river sleet Jul 16, 2024, 4:54 AM

#

me neither

#

the biggest thing is vaultwarden for me

past ravine Jul 16, 2024, 4:55 AM

#

have you tried out plane for task mgmt

#

i couldn't get it to work. bunch of cors errors

river sleet Jul 16, 2024, 4:55 AM

#

paperless, freshrss, linkding and arr*

past ravine Jul 16, 2024, 4:55 AM

#

i imported over 1.1k tasks to notion from jira

river sleet Jul 16, 2024, 4:56 AM

#

and many others that I can't remember off top of my head since I shut down the vm to restore 😆

past ravine Jul 16, 2024, 4:56 AM

#

lol

river sleet Jul 16, 2024, 4:56 AM

#

no haven't tried plane...you got a link?

past ravine Jul 16, 2024, 4:56 AM

#

btw atlassian is a royal pain to delete your account

past ravine Jul 16, 2024, 4:57 AM

#

river sleet no haven't tried plane...you got a link?

https://github.com/makeplane

GitHub

Plane

Easy, flexible, open source project management software - Plane

past ravine Jul 16, 2024, 4:57 AM

#

past ravine btw atlassian is a royal pain to delete your account

i just sent a gdpr deletion request or something to atlassian support

river sleet Jul 16, 2024, 4:58 AM

#

yea looks like a pain

past ravine Jul 16, 2024, 4:58 AM

#

plane?

river sleet Jul 16, 2024, 4:58 AM

#

no the Atlassian

past ravine Jul 16, 2024, 4:58 AM

#

oh and have you tried harbor docker registry? last i checked the docker container tried mounting the entire file system

river sleet Jul 16, 2024, 4:59 AM

#

I'm just at 5% in the restore. 🤦‍♂️

past ravine Jul 16, 2024, 4:59 AM

#

river sleet no the Atlassian

yea there's a bunch of things to cancel everywhere

#

just create a new vm

river sleet Jul 16, 2024, 4:59 AM

#

I could but it be much longer...I have about 30 containers

#

on that 1 vm

past ravine Jul 16, 2024, 4:59 AM

#

past ravine https://github.com/makeplane

with a weird install script and no prebuilt images

river sleet Jul 16, 2024, 5:00 AM

#

and another 20 on another server vm

past ravine Jul 16, 2024, 5:00 AM

#

yikes

#

last proxmox release i tried was like 7.4 so

river sleet Jul 16, 2024, 5:01 AM

#

it's just slow because I have my pbs backups stored in another NAS

#

you should give it another go...it is great

past ravine Jul 16, 2024, 5:01 AM

#

yea in def installing it when i get that baremetal

river sleet Jul 16, 2024, 5:02 AM

#

I have it setup to wake on lan to my PBS and does a nightly backup and then the PBS shuts down when done

past ravine Jul 16, 2024, 5:02 AM

#

but rn it's easy cause i can make a backup script in 300 lines of ruby which handles everything

#

And then sync the encrypted copy to onedrive

river sleet Jul 16, 2024, 5:03 AM

#

I just got a 4 line script in a cron job 😄

#

that turns on my pbs server and does the backup and shuts down

past ravine Jul 16, 2024, 5:03 AM

#

integrated pve backups ?

river sleet Jul 16, 2024, 5:03 AM

#

yes

#

one line to wake up the pbs

past ravine Jul 16, 2024, 5:04 AM

#

gitlab takes 20 seconds to load but my internet speed is 1.5gb up / down lol

river sleet Jul 16, 2024, 5:04 AM

#

one line to connect the pbs server

#

one line to backup

#

one line to shutdown

#

wow that is slow

#

where you located

#

I get 1gig u/d

past ravine Jul 16, 2024, 5:04 AM

#

canada

#

vms too

#

same region

past ravine Jul 16, 2024, 5:06 AM

#

river sleet one line to backup

~~which references a 500 lines shell file~~

#

trol

river sleet Jul 16, 2024, 5:07 AM

#

😆

past ravine Jul 16, 2024, 5:07 AM

#

there's one thing i miss

river sleet Jul 16, 2024, 5:08 AM

#

oh yea...other usefull stuff i use is homebride, scrypted, teslamate

past ravine Jul 16, 2024, 5:08 AM

#

forget what it's called

#

but it saves a complete backup of both the ram state and the disk state

river sleet Jul 16, 2024, 5:08 AM

#

what do you host?

past ravine Jul 16, 2024, 5:08 AM

#

i have an outdated ish list on https://sdko.org/homelab

river sleet Jul 16, 2024, 5:09 AM

#

past ravine Jul 16, 2024, 5:09 AM

#

oh yea it's intentional

#

uhh refresh

#

but for live stalking https://crt.sh/?q=sdko.org

crt.sh | sdko.org

Free CT Log Certificate Search Tool from Sectigo (formerly Comodo CA)

river sleet Jul 16, 2024, 5:11 AM

#

yea i see it now

past ravine Jul 16, 2024, 5:12 AM

#

then there's this thing i'm working on. outdated ss but https://img.sdko.org/u/pBZSoY.png

#

fun fact i made my personal site way before i even bought my new domain so it was just doing nothing for several weeks. i have a good dozen to dos to resolve

river sleet Jul 16, 2024, 5:13 AM

#

nice

#

I just self host stuff that I need to make my life easier

past ravine Jul 16, 2024, 5:14 AM

#

i'm gonna steal some guy's website

#

looks nice

#

instead

river sleet Jul 16, 2024, 5:14 AM

#

LOL

past ravine Jul 16, 2024, 5:14 AM

#

notice how there's no frontend technologies in /stack

river sleet Jul 16, 2024, 5:14 AM

#

yea...you're a backend dev

#

i see you have promtail, lokie, grafana

past ravine Jul 16, 2024, 5:15 AM

#

look at certificate transparency logs

river sleet Jul 16, 2024, 5:15 AM

#

I had that working and now it's broken....another thing to fix 🤦‍♂️

past ravine Jul 16, 2024, 5:15 AM

#

i need to fix it too

#

stopped logging docker for some reason

river sleet Jul 16, 2024, 5:16 AM

#

i was using it to monitor activities to see if I get weird people trying to hack my stuff

past ravine Jul 16, 2024, 5:17 AM

#

fail2ban

river sleet Jul 16, 2024, 5:17 AM

#

yea I have fail2ban setup on most my stuff

past ravine Jul 16, 2024, 5:17 AM

#

omg tailscale mobile auto connect is a pain to disable to disable vpn

#

it's the kind of thing you setup and forget

river sleet Jul 16, 2024, 5:18 AM

#

and I have geo locking

past ravine Jul 16, 2024, 5:18 AM

#

but i'm gonna build a web ui to see blocked ips

#

cause lol

past ravine Jul 16, 2024, 5:19 AM

#

river sleet and I have geo locking

i block [...] or (ip.geoip.country in {"AF" "BD" "CN" "IR" "IQ" "KP" "RU" "YE" "XX"}) or [...]

#

xx being other territories and nations

river sleet Jul 16, 2024, 5:20 AM

#

I also try to be secure as well...got vlans setup in my home network to segragate different things

past ravine Jul 16, 2024, 5:21 AM

#

idk if it's worth vlan'ing in my use case for tailscale stuff

#

public things are running in same rev proxy container so

#

different containers and i get tailscale intranets separate ips from the machine's so technically ig?

river sleet Jul 16, 2024, 5:22 AM

#

yea

past ravine Jul 16, 2024, 5:23 AM

#

100.64.0.0/10 is for my devices
100.65.0.0/10 is for my vms
100.66.0.0/10 is for my intranet ips so at least i organize

river sleet Jul 16, 2024, 5:23 AM

#

LOL I'm now at 26% restore...it's 107GB.

past ravine Jul 16, 2024, 5:23 AM

#

lol

#

how

river sleet Jul 16, 2024, 5:23 AM

#

yea you got it organized well

past ravine Jul 16, 2024, 5:24 AM

#

nice little notion docs

river sleet Jul 16, 2024, 5:24 AM

#

you are very organized

past ravine Jul 16, 2024, 5:24 AM

#

thanks

river sleet Jul 16, 2024, 5:24 AM

#

I need to put stuff down on paper one day...it's all in my head and sometimes I forget

past ravine Jul 16, 2024, 5:24 AM

#

wait no i inverser something in my message

#

trust notion not my words

past ravine Jul 16, 2024, 5:25 AM

#

river sleet I need to put stuff down on paper one day...it's all in my head and sometimes I ...

wait until you see my todo

#

#13 is still open

#

"Push docker images from < > to new registry" which is not necessarily hard to do

river sleet Jul 16, 2024, 5:28 AM

#

yea not hard

past ravine Jul 16, 2024, 5:28 AM

#

it's changing it in my ci/cd which is annoying

#

which leads me to tdk-12 get gitlab host name working as intended cause rn it only works as http://gitlab:80 as external url which makes my life pain

river sleet Jul 16, 2024, 5:30 AM

#

Are you doing this for learning for work?

past ravine Jul 16, 2024, 5:30 AM

#

kind of

#

but if i really wanted to learn i would switch to k8s

river sleet Jul 16, 2024, 5:32 AM

#

Yea...this is just my hobby

past ravine Jul 16, 2024, 5:32 AM

#

i might do something in cs

#

idk what yet but probably not software engineer

#

i won't say i hate writing code but i don't see myself doing that as a job

river sleet Jul 16, 2024, 5:34 AM

#

it's not a bad job though for the money

past ravine Jul 16, 2024, 5:34 AM

#

i mean- yea but still

#

do u like writing code

river sleet Jul 16, 2024, 5:36 AM

#

yea I do

past ravine Jul 16, 2024, 5:36 AM

#

what lang

#

and full stack or

river sleet Jul 16, 2024, 5:36 AM

#

I just do a lot of vb and sql

past ravine Jul 16, 2024, 5:37 AM

#

old school

#

vb is gonna be gone when tech debt is cleaned up in msft's codebase

#

replaced with python

river sleet Jul 16, 2024, 5:38 AM

#

yes I'm also old too..LOL

#

been in the field for almost 25 years

past ravine Jul 16, 2024, 5:39 AM

#

u work at ibm or something ?

river sleet Jul 16, 2024, 5:39 AM

#

🤣

past ravine Jul 16, 2024, 5:39 AM

#

real question

#

or for the us gov maintaining the last nuclear control centers with 8 inch floppy disks

river sleet Jul 16, 2024, 5:40 AM

#

no...just an old company that's been around for over 100 years

past ravine Jul 16, 2024, 5:40 AM

#

💀

#

canada gov isn't better

#

they use angular

river sleet Jul 16, 2024, 5:42 AM

#

at least they not using lisp

past ravine Jul 16, 2024, 5:43 AM

#

lol

#

my dad is sr project manager there so

#

i have the chance to see a lot of the inner workings

river sleet Jul 16, 2024, 5:44 AM

#

that's good you see it and know you don't want to do programming

#

I think I'm just going to call it a night and just let the damn restore go

past ravine Jul 16, 2024, 5:45 AM

#

same

river sleet Jul 16, 2024, 5:45 AM

#

It's going to be awhile

past ravine Jul 16, 2024, 5:45 AM

#

it's 1:45 am

river sleet Jul 16, 2024, 5:45 AM

#

I'll let you know if this works tomorrow

past ravine Jul 16, 2024, 5:45 AM

#

and i need to be up by 5 ~~today~~

river sleet Jul 16, 2024, 5:45 AM

#

dang you better get some rest

past ravine Jul 16, 2024, 5:45 AM

#

yea lol

#

cya

river sleet Jul 16, 2024, 5:46 AM

#

bye

river sleet Jul 18, 2024, 10:09 PM

#

@past ravine FYI...I spent too much time trying to figure this out. I ended up rolling my VM all the way back until before the issue occurred and so far it is working fine. I was on 2024.4.2 and pg12

past ravine Jul 18, 2024, 10:10 PM

#

did you try saving a snapshot of that again and then updating pg and authentik at the same time

river sleet Jul 18, 2024, 10:15 PM

#

not yet...I'm planning to do that tomorrow. 2 reasons. 1 is because I have a nightly backup that I want to capture a good state and 2 because I want to let 24 hours lapses to make sure it truly is resolved.

#

and third...I'm busy too. 😆

past ravine Jul 18, 2024, 10:16 PM

#

yea makes sense

#

i'm a little more intrepid i backup my docker data every 3h instead of the entire vm state

#

unless oracle and soon aws have ways of backing up entire vms without that costing money

river sleet Jul 18, 2024, 10:16 PM

#

that's intense

past ravine Jul 18, 2024, 10:17 PM

#

it's not too bad either

#

compressed around 3gb per vm (and encrypted ofc)

river sleet Jul 18, 2024, 10:17 PM

#

Yea mine is incremental backups so not long to do and not much storage

#

But that would mean I would have to leave my PBS on all the time which I don't want to due to high electricity cost

past ravine Jul 18, 2024, 10:17 PM

#

i'm actually rewriting my backup manager to rust to redeploy it easily

#

since you have been in the industry a while what do you think of my current setup: all my vm mgmt things are binaries which are loaded to a docker base image as needed

#

kind of like software updates like ios

river sleet Jul 18, 2024, 10:20 PM

#

I think that's good and probably fits your needs. It's little overkill for my purposes

past ravine Jul 18, 2024, 10:20 PM

#

well i feel it's the easiest to maintain and use

#

instead of having to maintain several containers there's just one with binaries loaded on them. like linux rn

#

but i'm always looking into new things

river sleet Jul 18, 2024, 10:22 PM

#

that's good...I just want things to work. I don't have the bandwidth right now to tinker

#

I depend on these apps I have running

river sleet Jul 22, 2024, 9:28 PM

#

@past ravine I updated the pg to 16 and then authentik to 2024.6.1 and it's been over 24 hours with no problem. Sometimes it's just easier to rollback than to figure out the issue. Thanks for all the help though.

#Backup Settings