Student having a hard time on Django deploy via GitLab CI | Django | Page 1

jade eagle Mar 1, 2025, 1:54 PM

#

Hello everyone,

I'm a student for a bachelor in development and conception. I chose Django for my school project and I have the obligation to deploy the app. I've decided to host a Debian server on Digital Ocean and make an automated deployment from GitLab CI.

As a former sysadmin in an old school company, I've never done this so this is my first time doing this and I'm having a hard time with several points.

passing variables securely.
deploying with Docker.
overall process and best practices.

What I have done already :

Linux server is running. Prerequisites OK. SSH OK.
GitLab variables OK.
Done dockerfile and gitlab-ci.yml. Could certainly be not fully correct...

Is there any kind soul here that could help a desperate student for his first (unsuccessful) try at deploying an app ? 🙂
I'm losing a bit of hope after... looks at GitLab history, almost 130 pushes... 😭

Best regards

Edit : sorry if my english is a bit lacking. This is not my first language.

split solstice Mar 1, 2025, 3:46 PM

#

So you are trying to build your Django project into a Docker image on GitHub, and then you want to automatically start/run/deploy that to your server?

jade eagle Mar 1, 2025, 4:22 PM

#

That's it. I build and push a docker image then want to pull it from my server. I don't even know if it's a good idea..

#

My repo is public as it's a student work that every teacher is supposed to be able to review.
https://gitlab.com/TakoyaKitten/cinephoria-ecf/-/tree/deploy?ref_type=heads

GitLab

Files · deploy · TakoyaKitten / Cinephoria-ECF · GitLab

GitLab.com

#

I have several questions about this. Deploying without docker, I've done. Cloning repo, creating a .env.prod file with the good variables which are mandatory to start the app then run. With Docker, eh... Harder. With automation, eh... Even more. I don't even understand how to pass the .env file. I don't want it to be on git so the password and all are safe. So I've created an env file variable on GitLab but I fail to use it in my pipeline

#

To be fair, I don't even know the best practices. The Django documentation is a bit... Hard to digest and most tutorials I've found are great on some points and bad at others. For example, the last I've followed, guy makes perfect explanation of the image creation, the push, the pull etc... And ends the tutorial by running python manage.py runserver on his remote server.... 🤨

split solstice Mar 1, 2025, 5:05 PM

#

Well, how i would do it:

You should be able to build a Docker image using your checked in code on your development machine, and be able to start it running/working manually.
Then you should start looking into making it auto-build on GitHub.
Then you setup Docker to start your image on your server, either with Docker directly, or via systemd/Docker compose.
Then i would use something like a watchtower/pyouroboros/ouroboros Docker running on the same machine to automatically pull down a new Docker image whenever you rebuild it on GitHub.

jade eagle Mar 1, 2025, 9:28 PM

#

Thanks I'll try. How do I manage the .env file in this setup ?

#

If the file is added to the docker image and pushed on docker repo, isn't it a security issue ?

split solstice Mar 1, 2025, 9:42 PM

#

You would supply the env vars in step 3, by i.e. spesifying the ENV vars directly in docker/a systemd service file/docker compose file.

jade eagle Mar 1, 2025, 9:45 PM

#

django-web:
    build: .
    container_name: django-docker
    ports:
      - "8000"
    depends_on:
      - db
    env_file:
      - .env.prod

#

done already

#

Right now I'm fighting with nginx. I don't understand why, impossible to make ti route to django app.
browser gives "ERR_EMPTY_RESPONSE"

#

I'm getting a bit mad right now xD

split solstice Mar 1, 2025, 9:46 PM

#

jade eagle ```yaml django-web: build: . container_name: django-docker ports: ...

Read https://discord.com/channels/856567261900832808/1306622915365703681 on howe to share your code (format) properly.

#

How does you nginc config look ?

jade eagle Mar 1, 2025, 9:47 PM

#

I know how to do it. Just forgot. edited

#

# Sets the max number of simultaneous connections that can be opened by a worker process
events {
   worker_connections 1024;
}

http {
   server {
       include mime.types;
       default_type application/octet-stream;
       sendfile on;
       keepalive_timeout 65;
       listen 80;

       # Requests to /static/ are served directly from the /static/ directory
       location /static/ {
           alias /static/;
           expires 7d;
       }

       # Configuration for serving media files
       # location /media/ {
       #     alias /home/app/web/mediafiles/;
       # }

       # Handles all other requests
       location / {
           # Forward requests to Django application
           proxy_pass http://django-web:8000;

           # Pass important headers to Django for proper request handling
           proxy_set_header Host $host;                          # Original host header
           proxy_set_header X-Real-IP $remote_addr;             # Client's real IP
           proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;  # Chain of IP addresses
           proxy_set_header X-Forwarded-Proto $scheme;          # Original protocol (http/https)
       }
   }
}

#

I temporarily added * to allowed hosts in my settings.py for debugging purposes. Still not working so I guess it's coming from nginx, though I could be mistaken

#

and this is the rproxy in docker compose :


  frontend-proxy:
    image: nginx:latest
    ports:
      - "8000:8000"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
      - ./static:/static:ro
      - ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
    depends_on:
      - django-web
volumes:
  postgres_data:

split solstice Mar 1, 2025, 9:51 PM

#

is this nginx for a server running on the host or in another container ?

jade eagle Mar 1, 2025, 9:51 PM

#

I tried with and without replacing the default.conf

split solstice Mar 1, 2025, 9:51 PM

#

right, in a container.

jade eagle Mar 1, 2025, 9:51 PM

#

yup

#

The fact that I have an "ERR_EMPTY_RESPONSE makes me think I have a routing problem between nginx and django or the two containers. But I'm noob so I'm not sure of anything and have a hard time finding the cause

split solstice Mar 1, 2025, 9:52 PM

#

yeah

#

i agree with the assumption

#

but i can not see anything abviously wrong.

jade eagle Mar 1, 2025, 9:53 PM

#

😭

split solstice Mar 1, 2025, 9:53 PM

#

jade eagle and this is the rproxy in docker compose : ```yaml frontend-proxy: imag...

You have pasted two sets of docker compose settings, but i assume they are all in one continious docker compose file ?

jade eagle Mar 1, 2025, 9:54 PM

#

services:
  db:
    image: postgres:17
    ports:
      - "5432:5432"
    volumes:
      - postgres_data:/var/lib/postgresql/data
    env_file:
      - .env.prod

  django-web:
    build: .
    container_name: django-docker
    ports:
      - "8000"
    depends_on:
      - db
    env_file:
      - .env.prod

  frontend-proxy:
    image: nginx:latest
    ports:
      - "8000:8000"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf:ro
      - ./static:/static:ro
      - ./nginx.conf:/etc/nginx/conf.d/default.conf:ro
    depends_on:
      - django-web
volumes:
  postgres_data:

#

docker container stack or whatever it is called

#

I also have this entrypoint, if it can help :

#!/usr/bin/env bash
echo "Starting Django entrypoint script..."
which python
pwd
ls -liah
python -m pip show django
python manage.py collectstatic --noinput
python manage.py migrate --noinput
# RETIRER LIGNE LOADDATAPOUR VRAI PROD #
python manage.py loaddata initial_data
echo "Starting Gunicorn server..."
python -m gunicorn --bind 0.0.0.0:8000 --workers 3 cinephoria.wsgi:application

#

In case gunicorn is implied in the issue

split solstice Mar 1, 2025, 9:56 PM

#

what is the log output from the django container ?

jade eagle Mar 1, 2025, 9:57 PM

#

2025-03-01 22:40:21 [2025-03-01 21:40:21 +0000] [13] [INFO] Starting gunicorn 23.0.0
2025-03-01 22:40:21 [2025-03-01 21:40:21 +0000] [13] [INFO] Listening at: http://0.0.0.0:8000 (13)
2025-03-01 22:40:21 [2025-03-01 21:40:21 +0000] [13] [INFO] Using worker: sync
2025-03-01 22:40:21 [2025-03-01 21:40:21 +0000] [14] [INFO] Booting worker with pid: 14
2025-03-01 22:40:21 [2025-03-01 21:40:21 +0000] [15] [INFO] Booting worker with pid: 15
2025-03-01 22:40:21 [2025-03-01 21:40:21 +0000] [16] [INFO] Booting worker with pid: 16

#

I have a lot of errors prior but they're about the fixtures loading finding already present data. Not blocking the django app start. Already tried from scratch with no data and I end with the same result as now

split solstice Mar 1, 2025, 9:59 PM

#

so how are you attempting to open the webpage in your browser? Exact url with port number please ?

jade eagle Mar 1, 2025, 10:00 PM

#

http://127.0.0.1:8000/

#

I use docker locally for now. Not on my remote debian yet. No need to deploy before it even works here

split solstice Mar 1, 2025, 10:02 PM

#

you are using port 8000 a lot of places

jade eagle Mar 1, 2025, 10:02 PM

#

cinephoria-project-db-1
postgres:17
5432:5432⁠

django-docker
cinephoria-project-django-web:<none>
60312:8000⁠

cinephoria-project-frontend-proxy-1
nginx:latest
8000:8000⁠

The docker desktop summary

small token Mar 1, 2025, 10:02 PM

#

Why are django-web and frontend-proxy using the same ports?

split solstice Mar 1, 2025, 10:02 PM

#

let's try to change this

#

  frontend-proxy:
    image: nginx:latest
    ports:
      - "8001:80"

#

and try http://127.0.0.1:8001/

jade eagle Mar 1, 2025, 10:03 PM

#

You know, when GPT gave me this I thought "weird to have the same port". Fuck me if it's right. Never trust it, I know

#

Ok. Running the build. Fingers and toes crossed

#

Ok, to things to say.

thanks a lot
brb, gonna press my gonads in a nutcracker

split solstice Mar 1, 2025, 10:06 PM

#

did it work?

jade eagle Mar 1, 2025, 10:06 PM

#

yes

#

hence my previous answer

split solstice Mar 1, 2025, 10:07 PM

#

skip 2, and rather remember to always doubdt what a AI tells you.

jade eagle Mar 1, 2025, 10:07 PM

#

now to test live. in prod env.

Just a side question :

is my allowed_host correct ?

ALLOWED_HOSTS = os.getenv("DJANGO_ALLOWED_HOSTS","127.0.0.1").split(",") + ["frontend-proxy"]

#

the DJAGO_ALLOWED_HOSTS variable has the digital ocean host ip

split solstice Mar 1, 2025, 10:09 PM

#

you probably want the digital ocean host ip + whatever domain name you point to the P and try to access the app using.

jade eagle Mar 1, 2025, 10:09 PM

#

split solstice skip 2, and rather remember to always doubdt what a AI tells you.

Oh believe me I hate it already. I mostly use it to assist in thinking and barely to code or else. I'm only beginner but i've already seen so much errors and BS from chatGPT I can't even understand how we're supposed to get replaced...
Thing's can turn in circle for hours to a point I'm in disbelief

split solstice Mar 1, 2025, 10:09 PM

#

+ ["frontend-proxy"] should not be required.

small token Mar 1, 2025, 10:09 PM

#

Maybe try reading the official docs first before asking ChatGPT

jade eagle Mar 1, 2025, 10:10 PM

#

split solstice you probably want the digital ocean host ip + whatever domain name you point to...

no domain name. It's a school project and it's not mandatory. I don't want to pay a domain name reservation.

split solstice Mar 1, 2025, 10:10 PM

#

split solstice ` + ["frontend-proxy"]` should not be required.

Since you are doing proxy_set_header Host $host; # Original host header, Django will never see frontend-proxy as a Host header value.

jade eagle Mar 1, 2025, 10:10 PM

#

small token Maybe try reading the official docs first before asking ChatGPT

I mostly do but right now I'm drowned in so many things I got lost ^^

jade eagle Mar 1, 2025, 10:10 PM

#

split solstice Since you are doing `proxy_set_header Host $host; # Ori...

noted

split solstice Mar 1, 2025, 10:11 PM

#

i am really happy AI did not exsist when i took my bachelors 😛 Wolfra Alpha did, and that already negatively affected my learning from math classes 😛

jade eagle Mar 1, 2025, 10:12 PM

#

I like Django a lot more than Symfony but Symfony had such a good documentation. I've never seen another one that good in any IT field.

split solstice Mar 1, 2025, 10:13 PM

#

ugh, PHP 😅

#

but yeah, looks like quite good docs.

jade eagle Mar 1, 2025, 10:14 PM

#

I would like to do without it Kag_ee but my formation is abysmally bad. Professors are nowhere to be found most of the time and resources are badly written or recorded. Students are not helping each other at all. I'm only there for the official degree it gives. And as I'm the only IT guy in my whole entourage, I feel a bit alone facing all this.

jade eagle Mar 1, 2025, 10:14 PM

#

split solstice ugh, PHP 😅

that's why I quit. I hated PHP. So bloated and complicated

#

rebooted my whole project.

#

Good side : I did in three weeks what I did in three months so I gained a bit of experience after all

split solstice Mar 1, 2025, 10:14 PM

#

But imho, i do Python becase Python is not "only web backend", as i feel PHP is.

jade eagle Mar 1, 2025, 10:15 PM

#

That's the main point of my redirection to python. In my project, I also have to do a desktop app and a mobile app. PHP didn't give me tools for this

split solstice Mar 1, 2025, 10:15 PM

#

you can (and other will not look at you weird) to write cli apps, web apps, program microcontrollers etc. as opposed to PHP.

#

however, PHP was the first language i learned.

#

Then i learned better after a few years 😛

jade eagle Mar 1, 2025, 10:16 PM

#

That's literally what I told my wife.

"you know, with python, I won't be only attached to web. It broadens the horizon of my possibilities"

#

yet I only versed in the web part for now. My school excam project first. Then the funsies

small token Mar 1, 2025, 10:18 PM

#

Wife? Aren't you a student? Here you are doing things in the wrong order again 😜

jade eagle Mar 1, 2025, 10:18 PM

#

I'm 35

small token Mar 1, 2025, 10:19 PM

#

Career switcher?

jade eagle Mar 1, 2025, 10:19 PM

#

I've been a sysadmin for 12 years already, even if it was for a very old school company with not much interesting stuff to do

#

yup

#

I have a sysadmin degree in France. Want a better degree in dev then another 2 year formation to be devops

#

It's my "before 40 goal"

#

It's not an easy moment, but i'm motivated and like to learn

#

It's just a matter of time

#

(and a bit of luck)

#

weird.. I have a Bad Request (400) now. It doesnt like my allowed host line.

Works again with a wildcard only

#

Well I'm gonna find a solution.

Once again, I thank you a lot for the time spent and the help provided. I was going insane.

Last question (or maybe noooot) :

Is it ok to mount an .env file provided on the host to void getting the .env in the docker image ?

Something like :

django-web:
    build: .
    container_name: django-docker
    volumes:
      - /$host/app/.env:/app/.env:ro
    ports:
      - "8000"
    depends_on:
      - db
    env_file:
      - .env.prod

I don't know if it's remotely possible. Would be cool 'cause you could have different.env for different clients and avoid security issues if I had to share the docker image etc etc

split solstice Mar 1, 2025, 10:36 PM

#

You should not have to, since you can use env_file 🤔

jade eagle Mar 1, 2025, 10:37 PM

#

yes but this imply I copy the env file in the image. If I could put it in dockerignore, and provide it manually ONCE one my production server by SSH it would assure my passwords are nowhere else than on my production server

split solstice Mar 1, 2025, 10:38 PM

#

huh ?

small token Mar 1, 2025, 10:38 PM

#

You probably don't want that in the image itself since different envs will have different values for its env vars

jade eagle Mar 1, 2025, 10:38 PM

#

That's why I'm asking this. SOrry english is a bit harder than my mother tongue

split solstice Mar 1, 2025, 10:38 PM

#

env_file is just read/used then the image is started / made into a running container, not when it is beeing built.

jade eagle Mar 1, 2025, 10:39 PM

#

oh. So it's never exported into the registry ?

#

I'm a bit lost. I had understood that when building the docker image, it adds everything not in the docker ignore, makes an image of it and push it into the docker registry. Then it's pulled on the remote debian I have

#

Did I understand something wrong ?

small token Mar 1, 2025, 10:58 PM

#

it adds everything not in the docker ignore

Yup. That's right. I'd suggest using dockerignore if you don't want it included in the build.

#

You don't have to make your own image. I think it's enough that you have a Dockerfile and config for docker-compose

jade eagle Mar 1, 2025, 11:00 PM

#

But that imply copying all sources on remote server then do a docker compose on it ?

small token Mar 1, 2025, 11:00 PM

#

Ah. Right. My bad.

jade eagle Mar 1, 2025, 11:01 PM

#

So mounting an env file as a volume is not possible ?

small token Mar 1, 2025, 11:01 PM

#

It's possible but you typically wouldn't want that. You'd want some other way to set your env vars

jade eagle Mar 1, 2025, 11:02 PM

#

So if it's not desired to add them to the image, not mounting it as a volume, how ?

I could try with the gitlab variables but last time, it was a nightmare and didn't work

small token Mar 1, 2025, 11:06 PM

#

So I typically do deploys in one of two ways:

keep things simple and just use docker-compose for small/toy projects
some platform-as-a-service that lets me add the environment variables in a form. Example: AWS Beanstalk or Heroku or Digital Ocean App Platform

jade eagle Mar 1, 2025, 11:06 PM

#

ok you know what ? It's a schoole project. In docker image it is

#

I'm fed up xD

small token Mar 1, 2025, 11:06 PM

#

Where are you deploying?

jade eagle Mar 1, 2025, 11:06 PM

#

Digital Ocean, debian

#

Thanks GitHub student pack

small token Mar 1, 2025, 11:07 PM

#

You might just be overcomplicating things, but I don't know what your needs/goals are.

jade eagle Mar 1, 2025, 11:07 PM

#

passing by a GitLab pipeline

#

Honestly ? They just want me to have an automated deploy when I push on main

#

And the lesson contains ZERO example / methodology. A bit of a "go schwing yourself" cursus imho

small token Mar 1, 2025, 11:08 PM

#

Digital Ocean Platform has push-to-deploy integration with GitLab and GitHub

jade eagle Mar 1, 2025, 11:08 PM

#

Is it accessible with only a gitlab student pack ?

#

Lot of functions are behind paywall

#

but honestly if I manage to make that damn allowed_hosts to work, it's perfect as it is right now

#

Security is an issue only if I share that image and as it's not truly in production and never will, it's a moot point

#

Is there a secret formatting ritual or something for the allowed hosts line ? I don't find a lot on the django doc and as soon as I put something else than

ALLOWED_HOSTS = "*" or ALLOWED_HOSTS="127.0.0.1", it doesn't work. When I use the os.getenv, I get a bad request 400

#

the .env.prod only contain the IP, without quotes or anything. And only one entry

small token Mar 1, 2025, 11:12 PM

#

Is it accessible with only a gitlab student pack ?

It's paid.

#

Have you tried hardcoding the IP just to make sure that it's not a problem with fetching the env var?

#

ALLOWED_HOSTS = "*" or ALLOWED_HOSTS="127.0.0.1", it doesn't work. When I use the os.getenv, I get a bad request 400

ALLOWED_HOSTS is a list of strings

jade eagle Mar 1, 2025, 11:20 PM

#

ALLOWED_HOSTS = os.getenv("DJANGO_ALLOWED_HOSTS","127.0.0.1").split(",")

in the .env.prod :

DJANGO_ALLOWED_HOSTS=207.154.206.57

Is it because it's not between quotes, it is not read as a string ?

small token Mar 1, 2025, 11:27 PM

#

It should look something like this in settings.py

ALLOWED_HOSTS = ['207.154.206.57', '127.0.0.1']

jade eagle Mar 1, 2025, 11:28 PM

#

ho. A tuple then. With strings inside

#

Well it's working locally. Can't test on the debian server yet, I have an issue. My pipeline fails because of "missing docker command" which is explained by the fact I only use python slim in my gitlab-ci.yml. I should add a docker image like docker:latest maybe. Gonna try

jade eagle Mar 1, 2025, 11:48 PM

#

well big failure in distant server.... I'm tired boss. 1 week without success. More than 150 commits... Ugh

small token Mar 1, 2025, 11:50 PM

#

What's the error?

jade eagle Mar 1, 2025, 11:59 PM

#

$ docker build -t $CONTAINER_IMAGE .
ERROR: error during connect: Head "http://docker:2375/_ping": dial tcp: lookup docker on 169.254.169.254:53: no such host
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1

#

It worked before, now it doesn't and I don't even know why.

#

OK I debugged it with unset DOCKER_HOST. Pipeline succeeded but no docker container running. Crashed at start

#

or never started

#

django.core.exceptions.ImproperlyConfigured: 'django.db.backends.None' isn't an available database backend or couldn't be imported. Check the above exception. To use one of the built-in backends, use 'django.db.backends.XXX', where XXX is one of:
'mysql', 'oracle', 'postgresql', 'sqlite3'

#

Doesn't read the env file

jade eagle Mar 5, 2025, 8:06 AM

#

Hello everyone.

I restarted the deployment from scratch and I finally made it work. But there is an issue :

django-docker  | 151 static files copied to '/app/staticfiles'.
django-docker  | Starting Gunicorn server...
django-docker  | [2025-03-04 18:19:15 +0000] [1] [INFO] Starting gunicorn 23.0.0
django-docker  | [2025-03-04 18:19:15 +0000] [1] [INFO] Listening at: http://0.0.0.0:8000 (1)
django-docker  | [2025-03-04 18:19:15 +0000] [1] [INFO] Using worker: sync
django-docker  | [2025-03-04 18:19:15 +0000] [8] [INFO] Booting worker with pid: 8
django-docker  | [2025-03-04 18:19:15 +0000] [9] [INFO] Booting worker with pid: 9
django-docker  | [2025-03-04 18:19:15 +0000] [10] [INFO] Booting worker with pid: 10
django-docker  | Not Found: /static/css/cinephoria.css
django-docker  | Not Found: /static/js/main.js
django-docker  | Not Found: /static/images/CinephoriaBannerLogo.png

My site is served without any static. CSS, JS and images.

my static are declared as follows :

STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, 'staticfiles')
STATICFILES_DIRS = [os.path.join(BASE_DIR, 'static')]

I'm using gunicorn. Static is in the same place than the rest of the project. What do I do wrong ?

#Student having a hard time on Django deploy via GitLab CI