Struggling with making test changes to deployment (added database configurations) | Django | Page 1

sonic shore Dec 18, 2024, 1:02 PM

#

Hey friends, I have a django app deployed on a ubuntu VPS with Postgres, Nginx, and Gunicorn.

I need to make some updates to the app, specifically I'm aiming to set up user logins, so I want to be uploading inputted user data to my postgres DB, but I'm struggling to figure out how I can set this up for testing locally before making the changes in production.

I have already got the django app in a git repo and the idea of making changes locally in vsCode to the styling CSS or whatever and then going back to the VPS to pull the changes is a workflow that makes sense to me. But I'm struggling with how I can access a copy of the VPS's postgres DB locally to play around with, if that makes sense?

Happy to answer clarifying questions - I'm super new to this so apologies if I don't make much sense

What I've tried:
I've tried allowing my local IP to connect to the VPS datbase by editing the postgresql.conf file and pg_hba.conf files in the VPS, but I get connection refused errors and I'm not sure why at this stage - I'm thinking maybe it would be easier if I just had a local "copy" of the DB instead to test with

raw fiber Dec 18, 2024, 1:05 PM

#

Installing a local copy is definitely the way to go. You don't really want to mess about with production data while running tests.
What OS are you developing on?

sonic shore Dec 18, 2024, 1:06 PM

#

I'm on Windows

raw fiber Dec 18, 2024, 1:07 PM

#

Damn 😅

sonic shore Dec 18, 2024, 1:07 PM

#

Does that make it very difficult?

#

or is it just that you're not on Windows

raw fiber Dec 18, 2024, 1:08 PM

#

It does for me as I haven't properly used it in over a decade 😁

sonic shore Dec 18, 2024, 1:08 PM

#

I see 😦

raw fiber Dec 18, 2024, 1:10 PM

#

I generally recommend using env vars for configuration, as I think they are the cleanest solution. You can set them on a per-unit basis assuming you deployed gunicorn via systemd.

#

If you install postgres locally, you can use something like https://django-classy-settings.readthedocs.io/latest/ to use env.dburl for configuration.

#

Or https://pypi.org/project/dj-database-url/ which is more specifically for database settings.

#

Then you can use the django admin https://docs.djangoproject.com/en/5.1/ref/django-admin/#dumpdata and loaddata to export data from your production server and load it locally

sonic shore Dec 18, 2024, 1:13 PM

#

i have installed postgres locally although it is a different version, does this matter?

raw fiber Dec 18, 2024, 1:14 PM

#

Assuming they aren't decades apart, probably not

sonic shore Dec 18, 2024, 1:14 PM

#

assuming you deployed gunicorn via systemd.
I'm not sure what systemd is tbh

raw fiber Dec 18, 2024, 1:14 PM

#

You wrote a .service file?

sonic shore Dec 18, 2024, 1:15 PM

#

yes I did

raw fiber Dec 18, 2024, 1:15 PM

#

Yeah, then you're using systemd 🙂

#

systemctl is a systemd management command.

sonic shore Dec 18, 2024, 1:16 PM

#

I see, I see

raw fiber Dec 18, 2024, 1:16 PM

#

You can read this when you have more time, if you're interested to know more: https://theepic.dev/books/lfd/linux_systemd/

sonic shore Dec 18, 2024, 1:17 PM

#

Okay cool

#

I will take a look at the resources you sent for now, I might ping you if I feel lost if that's okay

raw fiber Dec 18, 2024, 1:18 PM

#

Of course 🙂

#

If anything is unclear, do reach out 🙂

sonic shore Dec 18, 2024, 1:21 PM

#

I'm kind of wondering... Can it be as simple as me pulling the git repo locally and running python manage.py runserver ?

That would just mean that I'd need to make a local copy of the DB manually and then any changes to the DB I make locally, I would need to manually make them in the VPS DB too?

#

and for the rest, such as the login form/survey, I can just pull the code into the VPS using git?

#

I'm considering this because the DB structure only needs to be pretty basic

raw fiber Dec 18, 2024, 1:23 PM

#

You should be using python manage.py migrate to make changes go the DB (assuming you mean the schema, not data).

sonic shore Dec 18, 2024, 1:23 PM

#

Correct I meant schema

raw fiber Dec 18, 2024, 1:25 PM

#

Yeah, then you would edit your models locally, run python manage.py makemigrations, push those, and run migrate on the server.
It's a good idea to have it as a ExecStartPre in the service file.

sonic shore Dec 18, 2024, 1:28 PM

#

So I think where I'm lost is replicating the DB locally then...

I just made a clone of the VPS app files locally and then did manage.py runserver, but I get a long error ending with these lines

connection = self.Database.connect(**conn_params)
  File "C:\Users\Jamie\AppData\Local\Programs\Python\Python39\lib\site-packages\psycopg2\__init__.py", line 122, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: connection to server at "localhost" (::1), port 5432 failed: FATAL:  password authentication failed for 
user "admin"

Which I'm guessing is because I don't have a local copy of the DB set up?

raw fiber Dec 18, 2024, 1:28 PM

#

You need to configure permissions...

#

Did you create an admin user?

raw fiber Dec 18, 2024, 1:30 PM

#

raw fiber Yeah, then you would edit your models locally, run `python manage.py makemigrati...

Off the top of my head, so it's probably not going to work out of the box, but something like this:

[Unit]
Description=My Django app

[Service]
Environment="DJANGO_DB_URL=postgresql://myprojectuser:password@localhost/myproject"
ExecStartPre=/path/to/venv/bin/python manage.py migrate
ExecStartPre=/path/to/venv/bin/python manage.py collectstatic
ExecStart=/path/to/venv/bin/gunicorn [options]

sonic shore Dec 18, 2024, 1:30 PM

#

I think so I ran a command like ~/myprojectdir/manage.py createsuperuser

#

on the VPS

raw fiber Dec 18, 2024, 1:30 PM

#

createsuperuser creates a Django administrator. It's not the same thing.

#

You need to create a postgresql user for the database connection.

sonic shore Dec 18, 2024, 1:31 PM

#

ah yes, it was this command CREATE USER myprojectuser WITH PASSWORD 'password';

#

GRANT ALL PRIVILEGES ON DATABASE myproject TO myprojectuser;

#

and then I have the username and password in the django app settings.py file

#

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'myproject',
        'USER': 'myprojectuser',
        'PASSWORD': 'password',
        'HOST': 'localhost',
        'PORT': '',
    }
}

raw fiber Dec 18, 2024, 1:35 PM

#

What Django version are you on? And how did you install psycopg(2?)

sonic shore Dec 18, 2024, 1:39 PM

#

bare with, just trying to figure out how to check the version

#

how did you install psycopg(2?)
on the VPS, I created a virtual environment, activated it, and then ran pip install django gunicorn psycopg2-binary

#

I haven't installed it locally

raw fiber Dec 18, 2024, 1:42 PM

#

Ok. PsycoPG2 is older, and likely to be deprecated soon. If your Django version is new-ish, it's a good idea to replace it with psycopg-binary (version 3).

sonic shore Dec 18, 2024, 1:43 PM

#

django will be newish because I only set this up a couple weeks ago

raw fiber Dec 18, 2024, 1:43 PM

#

Also, as I said, hardcoding secrets in the settings file (including DB credentials) is generally a bad idea.

sonic shore Dec 18, 2024, 1:44 PM

#

okay, I'll add that to the todo list

raw fiber Dec 18, 2024, 1:53 PM

#

Updated the service unit above with an env var to set the DB URL.

If you use e.g. cbs, you can use something like this:

# settings.py
import os
from cbs import BaseSettings, env

DB_URL = os.environ.get("DJANGO_DB_URL", "postresql://devuser:insecurepassword@localhost/myproject")

class Settings(BaseSettings):
    def DATABASES(self):
        return {
            "default": env.dburl(DB_URL),
        }

Or check the docs for dj-database-url if you'd rather use that.
This would allow you to use devuser and insecurepassword as the local credentials, and set them securely in the service file.

willow vessel Dec 18, 2024, 1:57 PM

#

If you want to play with a copy of the production database, you can, (up to a number of rows where it is sloooow) use dumpdata and loaddata to copy all data fra prod to dev db 🤔

raw fiber Dec 18, 2024, 1:57 PM

#

raw fiber Then you can use the django admin <https://docs.djangoproject.com/en/5.1/ref/dja...

🐌 😁

willow vessel Dec 18, 2024, 1:57 PM

#

If you have media files in the DB you will also have to copy med media folder and have the same relative paths

willow vessel Dec 18, 2024, 1:58 PM

#

raw fiber 🐌 😁

Backlog was to long 😂

sonic shore Dec 18, 2024, 2:03 PM

#

Tbh I feel like I’ve deviated this a lot from what I’m trying to achieve, which I feel should be a pretty painless process

For now, I just want to get a copy of my VPS app working locally so I can start adding new features (e.g. login system)

I guess I’m just wondering what the easiest way to do this is?

#

Or is it just not that simple?

raw fiber Dec 18, 2024, 2:05 PM

#

While it may sometimes lead to bugs, you can always use sqlite in dev, and postgres in production.

#

The easiest way to replicate an identical setup is using Docker, tbh (but that's only easy if you spend days getting comfortable with docker).

raw fiber Dec 18, 2024, 2:07 PM

#

sonic shore ah yes, it was this command `CREATE USER myprojectuser WITH PASSWORD 'password';...

Did you create a DB user locally with the same credentials as in your settings file?

willow vessel Dec 18, 2024, 2:07 PM

#

Since both my prod and dev is using sqlite, i can just copy the database and media files directly.

sonic shore Dec 18, 2024, 2:10 PM

#

raw fiber Did you create a DB user locally with the same credentials as in your settings f...

Nope, all I’ve done locally is clone the project files and installed postgres

raw fiber Dec 18, 2024, 2:12 PM

#

Yeah, then the first step would be to create that user locally so Django can use the postgres DB

willow vessel Dec 18, 2024, 2:13 PM

#

(or just use sqlite for local dev)

sonic shore Dec 18, 2024, 2:14 PM

#

Whichever is easiest 😅

#

Or better - whichever im least likely to mess up

raw fiber Dec 18, 2024, 2:15 PM

#

Well, using sqlite for local dev would require at least setting the DB conditionally.

#

Which would also require setting DEBUG conditionally 🤔

#

Or using different env vars.

#

I think I'd go with setting up the postgres user

sonic shore Dec 18, 2024, 2:16 PM

#

Agreed!

#

And I only need the user, not to create a local database with the same name too?

raw fiber Dec 18, 2024, 2:17 PM

#

Not sure... it probably doesn't hurt to create the DB 🤔

#

I usually just use docker, so it's created automagically.... been a hot minute since I've used postgres on bare metal.

sonic shore Dec 18, 2024, 2:41 PM

#

Think I have progress....

So I set up a local DB and user

Server [localhost]:
Database [postgres]:
Port [5432]:
Username [postgres]:
Password for user postgres:
psql (17.2)
WARNING: Console code page (850) differs from Windows code page (1252)
         8-bit characters might not work correctly. See psql reference
         page "Notes for Windows users" for details.
Type "help" for help.

postgres=# CREATE DATABASE main;
CREATE DATABASE
postgres=# CREATE USER admin WITH PASSWORD 'x';
CREATE ROLE
postgres=# GRANT ALL PRIVILEGES ON DATABASE main TO admin;
GRANT
postgres=#

#

Then I ran python manage.py migrate

this gave me a big error ending

 raise MigrationSchemaMissing(
django.db.migrations.exceptions.MigrationSchemaMissing: Unable to create the django_migrations table (permission denied for schema publicLINE 1: CREATE TABLE "django_migrations" ("id" bigint NOT NULL PRIMA...
                     ^
)

#

BUT, if I run python manage.py runserver now, the app starts! Although it just says Bad Request (400)

raw fiber Dec 18, 2024, 2:47 PM

#

Is the name main what you have in settings?

#

Or is the name public? 🤔

sonic shore Dec 18, 2024, 2:51 PM

#

that is what I have in settings

#

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'main',
    'USER': 'admin',
    'PASSWORD': 'x',
    'HOST': 'localhost',
    'PORT': '',
    }
}

raw fiber Dec 18, 2024, 2:56 PM

#

Try this SQL:

GRANT ALL PRIVILEGES ON SCHEMA public TO admin;

🤔

sonic shore Dec 18, 2024, 2:57 PM

#

raise MigrationSchemaMissing(
django.db.migrations.exceptions.MigrationSchemaMissing: Unable to create the django_migrations table (permission denied for schema public
LINE 1: CREATE TABLE "django_migrations" ("id" bigint NOT NULL PRIMA...

same error :C

#

when running migrate

#

postgres=# GRANT ALL PRIVILEGES ON SCHEMA public TO admin;
GRANT

raw fiber Dec 18, 2024, 2:59 PM

#

Check python manage.py diffsettings for the line starting with DATABASES 🤔

#

It should show the correct settings, hopefully

sonic shore Dec 18, 2024, 3:01 PM

#

It did show a bunch of settings, was there anything you wanted in particular?

#

or just to check it works

raw fiber Dec 18, 2024, 3:03 PM

#

DATABASES

sonic shore Dec 18, 2024, 3:06 PM

#

DATABASES = {'default': {'ENGINE': 'django.db.backends.postgresql_psycopg2', 'NAME': 'main', 'USER': 'admin', 'PASSWORD': 'x', 'HOST': 'localhost', 'PORT': '', 'ATOMIC_REQUESTS': False, 'AUTOCOMMIT': True, 'CONN_MAX_AGE': 0, 'CONN_HEALTH_CHECKS': False, 'OPTIONS': {}, 'TIME_ZONE': None, 'TEST': {'CHARSET': None, 'COLLATION': None, 'MIGRATE': True, 'MIRROR': None, 'NAME': None}}}

raw fiber Dec 18, 2024, 3:10 PM

#

Hmmm, it seems you may need to create a new schema, and pass settings a search_path option if you're on Posgtres >= 15: https://gist.github.com/axelbdt/74898d80ceee51b69a16b575345e8457

sonic shore Dec 18, 2024, 3:14 PM

#

looking into....

#

So I ran the following commands in psql according to the thread:

-- Connect to mydb
\connect main;
-- Create a new schema with myuser as owner
CREATE SCHEMA myschema AUTHORIZATION admin;

ALTER ROLE admin SET client_encoding TO 'utf8';
ALTER ROLE admin SET default_transaction_isolation TO 'read committed';
ALTER ROLE admin SET timezone TO 'UTC';

#

then I re-ran ./manage.py migrate - but am getting the same error for now 🥹

raw fiber Dec 18, 2024, 3:25 PM

#

Did you update the settings with the options key?

sonic shore Dec 18, 2024, 3:29 PM

#

oooo I did not, thank u

#

oooomg

#

getting all the nice green OK's now !

raw fiber Dec 18, 2024, 3:32 PM

#

So much for "easiest and quickest" but we got there 😂

sonic shore Dec 18, 2024, 3:32 PM

#

I re-ran runserver but it still just says bad request though :\ was hoping that would fix it

raw fiber Dec 18, 2024, 3:32 PM

#

Enable DEBUG?

#

It's probably something like ALLOWED_HOSTS 🤔

sonic shore Dec 18, 2024, 3:33 PM

#

Yes, that's right!

#

DisallowedHost at /
Invalid HTTP_HOST header: '127.0.x.x:8000'. You may need to add '127.0.x.x' to ALLOWED_HOSTS.

#

so just a settings tweak hopefully, lemme try

#

okaaaaay 🙂 working

#

albeit none of the static files are serving

#

WOOOOOOOOOOOOOOOOOOOO

raw fiber Dec 18, 2024, 3:36 PM

#

Django doesn't serve static files if DEBUG is False 😅

sonic shore Dec 18, 2024, 3:37 PM

#

really? It has worked for me previously

#

with CSS and images

#

Oh wasn't there a command

#

python manage.py collectstatic

raw fiber Dec 18, 2024, 3:39 PM

#

It really shouldn't, but if you run runserver, you can set DEBUG=True and simply not commit that 🤔

sonic shore Dec 18, 2024, 3:42 PM

#

Yeah but don't get me wrong, even with DEBUG=False, the static files aren't servinng

#

so far I've tried python manage.py collectstaticbut that didn't resolve

#

but it confirmed that it can see the static folder atleast

raw fiber Dec 18, 2024, 3:44 PM

#

You are running runserver, and not gunicorn or something? 🤔

sonic shore Dec 18, 2024, 3:45 PM

#

oh yeah, I am just running manage.py runserver - is that not right? 😅

#

i thought that would be fine for local development

raw fiber Dec 18, 2024, 3:46 PM

#

Yeah, that's fine

#

And you see 404s listed in the console for static files, when you load the page?

sonic shore Dec 18, 2024, 3:49 PM

#

Yes correct, e.g:

[18/Dec/2024 15:42:42] "GET /static/styles/style.css HTTP/1.1" 404 179
[18/Dec/2024 15:42:42] "GET /static/images/home_icon.png HTTP/1.1" 404 179
[18/Dec/2024 15:42:42] "GET /static/images/colour_semantics_icon.png HTTP/1.1" 404 179
[18/Dec/2024 15:42:42] "GET /static/images/base_icon.png HTTP/1.1" 404 179```

raw fiber Dec 18, 2024, 3:50 PM

#

python manage.py findstatic styles/style.css?

sonic shore Dec 18, 2024, 3:50 PM

#

File path is certainly looking correct

sonic shore Dec 18, 2024, 3:50 PM

#

raw fiber `python manage.py findstatic styles/style.css`?

uh huh No matching file found for 'styles/style.css'.

raw fiber Dec 18, 2024, 3:51 PM

#

What do the static file settings look like?

sonic shore Dec 18, 2024, 3:51 PM

#

import os
STATIC_ROOT = os.path.join(BASE_DIR, 'static/')```

#

thats all

#

forgot a line

raw fiber Dec 18, 2024, 3:53 PM

#

No STATICFILES_DIRS?

sonic shore Dec 18, 2024, 3:53 PM

#

yes static_root, as above

#

no staticfiles_dirs

#

it works on the VPS tho 🤔

raw fiber Dec 18, 2024, 3:54 PM

#

You serve them via nginx?

sonic shore Dec 18, 2024, 3:54 PM

#

I do

#

    os.path.join(BASE_DIR, 'static'),
)```

#

will try adding this....

raw fiber Dec 18, 2024, 3:54 PM

#

sonic shore ```STATICFILES_DIRS = ( os.path.join(BASE_DIR, 'static'), )```

Won't work

sonic shore Dec 18, 2024, 3:55 PM

#

ah ok

raw fiber Dec 18, 2024, 3:55 PM

#

Can't have STATIC_ROOT in there 😅

#

What do the nginx settings look like for static?

sonic shore Dec 18, 2024, 3:56 PM

#

uhmm - I think I'm a bit silly

#

I've got the django app proj files cloned locally, but that doesn't include any nginx files/settings I think?

raw fiber Dec 18, 2024, 3:57 PM

#

Yeah, that's fine...

sonic shore Dec 18, 2024, 3:58 PM

#

oh - so u wanna see the settings from the VPS?

raw fiber Dec 18, 2024, 3:58 PM

#

More curious about how it looks in production

sonic shore Dec 18, 2024, 3:58 PM

#

okay okay

#

server {
    server_name little-talk.org www.little-talk.org;

    location = /favicon.ico { access_log off; log_not_found off; }
    location /static/ {
        alias /home/admin/littleTalk/static/;
    }

...

raw fiber Dec 18, 2024, 4:02 PM

#

Yeah, you should have static input directories, and a static output directory...

#

I guess we can worry about best practices later 😅

#

And skip committing another change 🤔

sonic shore Dec 18, 2024, 4:03 PM

#

https://tenor.com/view/oopsie-oops-oops-sorry-gif-8365536587490118065

Tenor

raw fiber Dec 18, 2024, 4:04 PM

#

STATICFILES_DIRS = (
    os.path.join(BASE_DIR, 'static'),
)

Add this, but delete the STATIC_ROOT setting 🤔

#

Then don't commit settings.py to git 😅

#

(That's a good reason to use CBS, and have different settings for different environments :p)

sonic shore Dec 18, 2024, 4:05 PM

#

 python manage.py findstatic styles/style.css
Found 'styles/style.css' here:
  C:\Users\Jamie\Documents\GitHub\littleTalk-VPS\static\styles\style.css``` 
😏

#

oh interesting... still doesn't work

"GET /static/images/speech_therapy.png HTTP/1.1" 404 179
[18/Dec/2024 16:05:46] "GET /static/images/colour_semantics_icon.png HTTP/1.1" 404 179
[18/Dec/2024 16:05:46] "GET /static/images/stock_graphs.jpg HTTP/1.1" 404 179
[18/Dec/2024 16:05:46] "GET /static/images/learning_level.png HTTP/1.1" 404 179
[18/Dec/2024 16:05:46] "GET /favicon.ico HTTP/1.1" 404 179

raw fiber Dec 18, 2024, 4:07 PM

#

Try findstatic on one of those? 🤔

#

(Ignore favicon.ico)

sonic shore Dec 18, 2024, 4:07 PM

#

findstatic works ok

raw fiber Dec 18, 2024, 4:08 PM

#

(╯°□°)╯︵ ┻━┻

sonic shore Dec 18, 2024, 4:09 PM

#

it's gonna be okay ┬─┬ノ( º _ ºノ)

raw fiber Dec 18, 2024, 4:09 PM

#

DEBUG still True?

sonic shore Dec 18, 2024, 4:09 PM

#

nopee

raw fiber Dec 18, 2024, 4:09 PM

#

Oh, that be why :p

#

Django only serves static when DEBUG is True

sonic shore Dec 18, 2024, 4:09 PM

#

ehh??

#

Oh THATS what u meant

#

i thought u meant the other way round

#

AWWWWWW SHIT

#

we're up and running 🎊

#

https://tenor.com/view/confetti-style-gif-19616552

Tenor

#

I can't thank you enough Pat 😅

#

Days of me going crazy solved in a matter of hours

#

My steplist when deploying to prod:

`Local:
python manage.py makemigrations (if there is local DB changes)
Commit local changes (NOT SETTINGS.PY)

VPS:
Git pull
python manage.py migrate (if I have migrations)
python manage.py collectstatic (if I have new statics)`

raw fiber Dec 18, 2024, 4:20 PM

#

Sounds right... and restart the gunicorn server, of course

#

Though we'll need to fix your static file settings one of these days 😅

sonic shore Dec 18, 2024, 4:22 PM

#

I'll do it next

sonic shore Dec 18, 2024, 4:22 PM

#

raw fiber Yeah, you should have static input directories, and a static output directory...

u mean this, right?

raw fiber Dec 18, 2024, 4:24 PM

#

Which will also mean updating nginx settings, and comitting some changes to settings.py

sonic shore Dec 18, 2024, 4:25 PM

#

Okay... I'm feeling confident that I can have a go myself now that the ball is rolling! 🙂

#

random question, whenever I runserver etc, git wants to commit loads of changes on a binary level - is this normal?

#

I couldn't really find much online about it

raw fiber Dec 18, 2024, 4:27 PM

#

See my previous suggestions of keeping static in /srv/http/static/ (or /srv/http/littletalk/static/ if you have multiple projects)
(In STATIC_ROOT and nginx.conf)

#

What files?

sonic shore Dec 18, 2024, 4:28 PM

#

for example: littleTalkApp\__pycache__\urls.cpython-39.pyc

#

im using git desktop and it says "the binary file has changed"

raw fiber Dec 18, 2024, 4:28 PM

#

Yeah, add **/*.pyc to .gitignore, and run git rm on any that are already committed.

#

Those are python bytecode files... they should not be committed 🙂

sonic shore Dec 18, 2024, 4:29 PM

#

okay, good thing I checked then 😛

raw fiber Dec 18, 2024, 4:29 PM

#

https://theepic.dev/books/lfd/git_django/#keeping-our-repository-clean

#Struggling with making test changes to deployment (added database configurations)