Handle Trakt and MAL authentication with .env and public git repository

[dire sky]

Welcome @worthy sonnet!

Someone from <@&938443185347244033> will assist when they're available.

Including meta.log from the beginning is a huge help. Type !logs for more information.

After attaching your log, do not forget to hit the green check boxes when prompted by our bot.

You can press the "Close Post" button above or type /close at any time to close this post.

[worthy sonnet]

Wtf, why did it deleted my message ?

I think there is something in my YAML that the bot is preventing, I will try with a screenshot instead

Original message:

Hey, it's me again 😅

I have my Kometa repository in public and I was wondering how do you handle Trakt and MAL authorizations with git.

I've set the following when I created my config so I don't leak my access/refresh tokens (See screenshot)

It works well for the first initialization.
Kometa replaces the authorization block correctly for these services but it creates a conflict with git when I update my config.yml.

I was wondering how do you handle this if you host publicly your config.

Maybe there is a way to mitigate this with a new yaml file like authentication.yml that should be imported somehow in the config.yaml so when I modify my config on git, it doesn't interract with the config.yaml part that kometa replaces locally ?

[proper star]

yea the bot prevents people from exposing their api keys so anytime it detects those lines from the config it auto deletes the message

you can use kometa secrets btw

https://kometa.wiki/en/latest/kometa/environmental/?h=secrets#docker-environment_30

[worthy sonnet]

I already use them with a .env file.

My issue is that Kometa will use those secrets and replaces the config.yaml values of the following blocks :

trakt:
  authorization:
    access_token:
    expires_in:
    refresh_token:
    created_at: 

mal:
  localhost_url:
  authorization:
    access_token:
    expires_in: 
    refresh_token: 

So on my server, the config.yaml files become de-synchronized with git.
If I modify my config.yaml file on git and then pull the new config, it tries to use the values in the .env and those values aren't updated.

A workaround I can think about is to move this part of the config in an other file that I don't need to modify often so I can modify my config.yaml without having authentication issues when I pull a changed versions of the settings from git

[proper star]

you could also add an if statement into your script before the pull that checks if the repo is out of sync. something like

status="$(git status --branch --porcelain)"
message="Update $date"
if [ "$status" == "## main...origin/main" ]; then
  :
else
  git add .
  git commit -m "${message}" 
  git push https://git.servernation.xyz/anonfawkes/Kometa-Stuff.git
fi

I have a whole script that checks for git changes out of sync then pushes before my kometa run as I host all my kometa files on my own git repo

and sends discord webhooks with the commit message

[worthy sonnet]

But as Kometa changes automatically the config.yml file, It will always be out of sync ???
Or am I missing something here ?

[proper star]

your pulling before the run right?

[proper star]

if your pulling before, then this could go after the kometa run has finished to push to git

personally i have my git setup like this https://git.servernation.xyz/anonfawkes/Kometa-Stuff

I have a "fake" config.yml

anytime i make changes to my main config I just add them to that

and removed all the stuff that can leak

[worthy sonnet]

I don't understand, I don't want to push my credentials to git as my repo is public

[proper star]

but your using secrets no?

[worthy sonnet]

Yes I use secrets but only for the initialization as Kometa will use those secrets and replaces the config.yaml values of the following blocks :
trakt:
authorization:
access_token:
expires_in:
refresh_token:
created_at:

mal:
localhost_url:
authorization:
access_token:
expires_in:
refresh_token:

[proper star]

or are you saying that secrets is broken with these new trakt changes that happened and its autofilling them into the config

because it shouldnt be afaik

[worthy sonnet]

I will try to reformulate lol

[proper star]

i think i understand

[worthy sonnet]

yes

[proper star]

you use secrets and the lines in config are basically variables, then kometa runs and it autofills with the new "secrets"a and is replacing those variables with the actual keys and whatnot

[worthy sonnet]

exactly

[proper star]

hmmmm. i dont think that is supposed to happen with secrets. ill talk to the team and see but you could run carrots trakt/mal online auth locally (the one from the wiki) to print the auth to a separate file and use that file as your "secrets" by concatenating it into the run command i would imagine.

https://github.com/chazlarson/Media-Scripts/blob/main/Kometa/kometa-trakt-auth.py

https://github.com/chazlarson/Media-Scripts/blob/main/Kometa/kometa-mal-auth.py

[worthy sonnet]

Maybe there is a way to mitigate this with a new yaml file like authentication.yml that should be imported somehow in the config.yaml so when I modify my config on git, it doesn't interract with the config.yaml part that kometa replaces locally ?
As I shouldn't modify a lot auth, using a authentication.yml file shouldn't cause issues

[proper star]

trakt is every 24 hours now

@verbal ocean thoughts since I see your online

[worthy sonnet]

Yeah but as long as I don't modify authentication.yaml on git, it shouldn't have any issues git pulling and my credentials should be safe

[proper star]

TLDR is that he is using secrets but when kometa replaces the trakt auth in the config its causing the secrets to no longer be secret by putting them into his config and throwing git issues due to being out of sync

[proper star]

but youll run ito the same issue as kometa would be modifying it

trakt reauths every 24 hours. it will modify the expires_in and created_at at the very least

[worthy sonnet]

Yes but as long as I don't push the 'auth.yml' file modified on my server by Kometa, I'm good
And as long as I don't modify it on git, pulling shouldn't be an issue

My current problem is that I modify often the config.yaml, so every time I modify it on git, I need to reconfigure the auth on my server

[opal wharf]

This is not something that is built into the tool, and YAML does not have an "include" facility, so this isn't a way around this.

[opal wharf]

The only way around this at present is to either not check in your active config.yml or script around replacing these variables before you check it in.

Or write a script that supports this "auth.yml" notion by merging these files prior to the run.

You could use a pre commit task to do the variable replacement stuff on commit.

Or just wait for it to settle down before putting it on GitHub.

[worthy sonnet]

more like an action after pull, the push side from my main computer is fine
I use git on my server with a deploy_key so I just need to pull from git instead of doing some scp shenanigans to deploy my config haha

But yeah, I see what you mean
Weird that I'm the first person to have problems with this

[opal wharf]

This is just not a use case that the tool was designed for. It's also bad version -control practice generally.

Anyone who tries to version control their active config file and check it in after every run will probably see this.
The typical way to deal with it is to not version control the active config.

[opal wharf]

One problem here is you attempting to use secrets for the access and refresh tokens. Those are ephemeral, generated on demand as required, and are only written to config.yml. There's no mechanism for writing them through to wherever you are storing the secrets.

So you could:
Use secrets for those values, then after the run:

1. capture the current values from the config.yml
2. replace those values with your secret placeholders
3. write the new values into your .env or whatever
4. check in the config.yml. so you're never checking any changing values into your repo

If you do this you will want to use secrets for the other two ephemeral values as well; expires_in and created_at to ensure that the renewal happens as required.

The actual "solution" to this is to not try to version-control your active config.yml.
#general-chat message

[worthy sonnet]

defeats a bit the version-control imho

For now, what I do when I change my config file on git is the following :

• Connect to my server
• Backup my config file in an other folder so I have the acces_token config and refresh token
• git stash on my server so I have no longer changes vs git on my server
• git pull my new config
• copy manually the values from my backup to my new config

[opal wharf]

I don't understand how not checking in one particular file which contains sentive and ephemeral data "defeats the version-control"

[worthy sonnet]

Because it's the main file I modify to change things in kometa ??

[opal wharf]

That active config file should be treated as ephemeral and disposable. It shouldn't be something you check in because human error may expose sensitive data by doing so.

[worthy sonnet]

I think authorization block shouldn't be in the config.yml because kometa replace the values (maybe it should be in the config.cache or smthg like this)

[opal wharf]

The one that is checked in should be totally free of such things

[opal wharf]

I don't disagree, but at the moment it is.

[worthy sonnet]

I'm not human, don't worry haha 🤖

[worthy sonnet]

Yeah so at the moment i'm using the following workflow because I configure kometa on my PC and run it on my server 😉

For now, what I do when I change my config file on git is the following :
Connect to my server
Backup my config file in an other folder so I have the acces_token config and refresh token
git stash on my server so I have no longer changes vs git on my server
git pull my new config
copy manually the values from my backup to my new config

I don't run kometa on my pc so it doesn't override things

[opal wharf]

The right answer here is to check in and version control a template of some sort.

Then you have some [ideally automated] syncing process that writes changes you make to the active config back to the template, or better you make your changes in the template and have some automation that syncs those changes to the active.

[opal wharf]

That's all a lot of fuss that could be avoided by not checking your active config.yml into version control and instead using a template.

But you do you.

[worthy sonnet]

I agree, maybe I will go this way if decide to CI/CD my kometa with ansible, I will probably use a jinja2 template for this, but for now, I will stick with copy/pasting unless I have 4hours to lose for CI/CD stuff haha

[worthy sonnet]

I think it may be ok to leave this one open as the issue is still here
I will take a look at kometa code to see if I can pr smthg

[rare spindle]

Ok. Just don’t think the issue is fixable with current code…

[worthy sonnet]

Is there any guidelines about contributing ? Fork and create a PR ? Any tips about the sqlite db ?

[opal wharf]

Standard stuff, fork and PR into nightly. What sort of sqlite tips?

[worthy sonnet]

Anything I should know about or idk
Just looking at the code, it seems pretty doable

[opal wharf]

There's nothing remarkable about this sqlite DB conpared to any other so far as I know.

Of course it's doable, no one has said it isn't. It just touches a lot of documentation and walkthrough stuff; there's migration to consider, it could be that third-party things that leverage Kometa's config will break, etc.

[worthy sonnet]

Yes I understand, don't worry

Let me cook 😎

[worthy sonnet]

nice

[rare spindle]

Is that progress?

[worthy sonnet]

Yes I'm cleaning my code right now, trakt is ready to be stored in database with a setting in the config on my branch

Need to work on MAL

[rare spindle]

And will the daily refresh write the new token back?

[worthy sonnet]

I;m adding a setting store_authorization_in_db: true|false under settings in config.yaml
If this is set to true, trakt will ask to login with pin and store the authorization block in the database and you can then remove the authorization block from your yaml

I did it like this so it's recompatible with current settings

[worthy sonnet]

Can't make it work on MAL 😭
| Token endpoint response: {'error': 'invalid_grant', 'message': 'The provided authorization grant (e.g., authorization code, resource owner credentials) or refresh token is invalid, expired, revoked, does not match the redirection URI used in the authorization request, or was issued to another client.', 'hint': 'Failed to verify code_verifier.'} |

Is this a MAL oauth problem ?
even using this script gives me an error

[worthy sonnet]

I don't know why but can't get master branch auth to MAL, I will check tomorrow for this service

Just submitted a WIP draft PR if someone wants to take a look at it https://github.com/Kometa-Team/Kometa/pull/2655

[worthy sonnet]

Niceeee

[worthy sonnet]

https://github.com/Kometa-Team/Kometa/pull/2655 PR is ready for review

[sinful thistle]

I don't know if I like it being referred to as the db

We have consistently called it the cache file

I am assuming that's where we are talking about it being saved to?

I get it's a db, but that's not a terminology we've used for it anywhere in the docs

[worthy sonnet]

yes

cache works for me

[opal wharf]

Curious what happens if the user has a valid authentication block in the config and they flip this switch

[sinful thistle]

And has it been tested if the user deletes their cache file

I'm assuming yes but just want to be sure

[worthy sonnet]

yes i tested it

[worthy sonnet]

it ask for reauth

[opal wharf]

That seems suboptimal. Why not use the existing values and move them to the cache?

[sinful thistle]

So just to play devil's advocate, why not have a use_cache type setting for all connectors? Why single out Trakt and MAL?

[opal wharf]

These are the only two that write stuff back to the config.

[sinful thistle]

Ah gotcha

I haven't read the backstory here

[opal wharf]

The desire is to version-control the active config file. Kometa updating the file gets in the way of this.

[sinful thistle]

I guess because you're telling it not to use the cache, so why would it then go check the cache to move the info out. It's an additional check on every run

[opal wharf]

No, I'm talking about the case where someone has active working authentication data in the config and they tick the box that says "store this in the cache".

As a user, I woudl expect it to move my existing working auth data to the cache, not ask me to reauthenticate

[sinful thistle]

You would also need to code in the reverse of moving stuff from config into cache and back if the user is flipping the switch

[opal wharf]

Of course

Both seem pretty straightforward.

[sinful thistle]

There was a discussion at one point about having a secrets or env file with connector stuff

I think that was fairly recently

It was a user discussion rather than an internal discussion I'm sure

[opal wharf]

That would be another way to handle this. Put all keys in a second file.

[worthy sonnet]

i've renamed it in cache instead of db 🙂

[sinful thistle]

Would it be a "better" solution?

Maybe that's a personal opinion thing

[rare spindle]

Yes and then you could have protocol? The variable needs to be named <<Kometa_PlexToken>> etc?

[opal wharf]

One advantage of the separate file might be that it's just loading a second file which is either there or not, and if it is, then Kometa knows to write ephemeral keys to it.

[sinful thistle]

It would also mean that disabling the cache or deleting it wouldn't prompt for reauth

[rare spindle]

I like the idea of a separate file for that reason

[worthy sonnet]

It would be an other way to do this but why are you against putting it in the cache database?

[opal wharf]

It's ultimately up to the author which method he wants to go with if any.

[worthy sonnet]

A reason I see it's that it's harder to see what's in the sqlite for a standard user

[opal wharf]

I kind of like the separate file since it provides a pretty transparent way to move anything over there without having to maintain database access code for each thing.

[proper star]

I agree I also like the second file. Most projects seem to do a .env anyway or just use Dockers built-in secrets which you can use something like bitwarden to store those secrets. Also even though the other connections don't change if we're going to do trakt and Mal, we might as well just do them all

And dockers secrets are stored in a separate file that have certain restricted permissions on them and either your run command or your compose file just calls those files using the secrets variable

Basically like this

services:
  server:
    image: 'vaultwarden/server:latest'
    container_name: vaultwarden
    environment:
      - DOMAIN=/run/secrets/domain
      - ADMIN_TOKEN=/run/secrets/admin_token
      - INSTALL_ID=/run/secrets/install_id
      - INSTALL_KEY=/run/secrets/install_key
      - PUSH_ENABLED=true
      - SIGNUPS_ALLOWED=true
      - INVITATIONS_ALLOWED=true
      - TZ=America/New_York
    volumes:
      - './data:/data/'
    ports:
      - '8010:80'
    restart: always
    labels:
      - "diun.enable=true"
    user: 1000:1000
    secrets:
      - admin_token
      - install_id
      - install_key
      - domain
    networks:
      - localbridge
      - vaultwarden_proxy
secrets:
  admin_token:
    file: ./admin_token.txt
  install_id:
    file: ./install_id.txt
  install_key:
    file: ./install_key.txt
  domain:
    file: ./domain.txt
networks:
  localbridge:
    external: true
  vaultwarden_proxy:
    driver: bridge
    name: vaultwarden_proxy

https://docs.docker.com/engine/swarm/secrets/#build-support-for-docker-secrets-into-your-images

[worthy sonnet]

So what do you want to do with the PR ?

[opal wharf]

That's up to @verbal ocean

[weak bone]

Just to add my perspective as someone encountering the Trakt 24h token refresh issue in a gitops'd environment:

I'm liking the idea of storing (just) these ephemeral tokens within the cache as that already exists, but would need a method for seeding/migrating to it. Kometa could check if the config file's auth block is newer than what's in the database and copy it over initially, then rely solely on the database for subsequent operations.

Many of us are already using environment variables or docker secrets in our setups. In my kubernetes deployment, I'm pulling secrets from 1Password, but this is currently one-directional - there's no way for updated tokens to be saved back without adding in a separate container that monitors the loaded config and updates my secret store with changed tokens via a rather extensive script. Due to my config being loaded from a configmap, whatever secrets/envars are stored are added to the config at container start.

Simply reading and writing ephemeral tokens to a separate authorizations file doesn't fully solve the problem, but it would mean our workarounds wouldn't need to be as in depth I guess. I could then just have an init container that checks if the authorization file exists, create it, and populate it with the variables from secrets. Or if it exists already, check if the token in secrets is newer (mal might need some sort of created_at value...?).

Either way, with all that, I'd support @worthy sonnet's suggestion of a separate DB/cache as well as just separating the authorization sections from the main config file.