#django-storages with sftp, django v3.2

1 messages Β· Page 1 of 1 (latest)

hybrid widget
#

You can connect to the sftp server and upload files successfully?

gleaming shore
#

you mean by command line ?

hybrid widget
#

The method doesn't matter. I mean, can you connect to the sftp server and upload files without any error message?

gleaming shore
#

yes I can connect to it and upload files by those settings on my local environemt.

hybrid widget
#

hmm.. well that's strange. Can you see those files on the file system on the server?

gleaming shore
#

yes, but when I add the same settings on LIVE, it crashes, I will attach a screenshot of whats happening on production

hybrid widget
#

I remember having a custom paramiko based server at a previous job, and that was tons of super custom code. How do you do it?

gleaming shore
#

okay on local I have those settings:

USE_NNUH = config('USE_NNUH') == 'TRUE'
if USE_NNUH:
    # Static files settings
    STORAGES = {
        "default": {
            "BACKEND": "storages.backends.sftpstorage.SFTPStorage",
            "OPTIONS": {
                "host": "cdn.fake.org",
                "root_path": "/var/www/html/media/",
                "params": {'username': 'fake', 'password': 'fake', 'allow_agent': False},

            },
        },
        "staticfiles": {
            "BACKEND": "django.contrib.staticfiles.storage.StaticFilesStorage",
        }
    }
    MEDIA_ROOT = '/var/www/html/media/'
    MEDIA_URL = 'http://cdn.fake.org/media/'
else:
    # Local development settings
    STATIC_URL = '/static/'
    STATIC_ROOT = os.path.join(PROJ_DIR, 'static_collected/')
    STATICFILES_DIRS = [BASE_DIR / 'static', ]

    MEDIA_URL = '/media/'
    MEDIA_ROOT = os.path.join(PROJ_DIR, 'media/')

    STORAGES = {
        "default": {
            "BACKEND": "django.core.files.storage.FileSystemStorage",
        },
        "staticfiles": {
            "BACKEND": "django.contrib.staticfiles.storage.StaticFilesStorage",
        }
    }

#

those previous settings are work 100% correctly, using django-storages[sftp], django 4.2 , paramiko, bcrypt==3.2.0, cryptography==41.0.7

#

I'm able to upload and even read the files on django admin side: when I click any image ( that I have uploaded) it open in fake.nnuh.org, and I even see the image in the sftp media folder

#

on PROD I have django 3.2 :/

hybrid widget
gleaming shore
#

no actually I have not setup sentry on PROD, I'm relying on what debug=True shows

hybrid widget
hybrid widget
#

I don't understand the problem anymore. You can upload, you can view the files... what was the problem?

gleaming shore
#

okay: what I have posted yesterday is on PROD env.
and I'm saying to you that the same settings work on local env. !!
everything works correctly on local, but on prod ( its a production but not public until now) so no worries if I make DEBUG=True...
Do u understand me now ?

fringe briar
#

You have django 3.2 right?

gleaming shore
#

yes, on PROD, and I cannot do upgrade due to many reasons... do u think the version is a problem ?

fringe briar
#

So the problem is that your backend is not even connecting with your object storage right?

gleaming shore
#

yah and it appear like it doesn't see the sftp at all

fringe briar
#

I think it can be because this:
STORAGES = {
"default": {
"BACKEND": "storages.backends.sftpstorage.SFTPStorage",
"OPTIONS": {
"host": "cdn.fake.org",
"root_path": "/var/www/html/media/",
"params": {'username': 'fake', 'password': 'fake', 'allow_agent': False},

        },
    },
    "staticfiles": {
        "BACKEND": "django.contrib.staticfiles.storage.StaticFilesStorage",
    }
}

since this is the way that you declares the storages on django 4.2 or superior, for <4.2 you need to do it in this way:

DEFAULT_FILE_STORAGE = "storages.backends.s3.S3Storage"

STATICFILES_STORAGE = "storages.backends.s3.S3Storage"

gleaming shore
#

okay I will try to define them as u said and return back with the result

fringe briar
#

Ok

#

dont libs., is storages.

#

mb

gleaming shore
#

I know.

hybrid widget
#

If you have a crash, you should post the traceback and error message. It almost always contains a ton of very useful information for figuring out the problem.

I noticed also that Re had to guess what the problem was, and you answered yes. It would have been much better had you stated the problem.

Also you started by saying "it doesn't write the file to the sftp server" which then later was not true. Was that because you managed to fix something in the meantime and didn't communicate to us?

#

Getting help is about communicating clearly.

fringe briar
#

That's what I dont understand, there is an error or not?

hybrid widget
#

But it was also stated that files are not written to the server, and that files ARE written to the server.

fringe briar
#

Yeah but if it is written files, the problem it might to be because the storages but on that message @gleaming shore is saying the app crashing but, if it crash what is the output of the error?

gleaming shore
#

okay:

  1. I'm sorry If I make any misunderstanding to you guys
  2. the post ( originally) talk about my problem on production which is based on django v3.2
  3. what I print today is the settings that I have them locally for django v4, I have tried the settings locally and upload an image and I'm able to view the image successfully from the url of the sftp server, and I have check the sftp server, it has the image that I have uploaded ------ LOCAL THIS IS LOCAL
  4. lets talk about PROD -_-, on production I have change the conf to make it suitable for django 3.2 but still when I upload image it shows like corrupted image and the link of the image ( from inspect element) is domained on the website itself not from cdn. ( that's what I mean by crashes, I don't see any error/big error)
#
USE_NNUH = os.environ.get('USE_NNUH') == 'TRUE'
if USE_NNUH:
    # Static files settings
    DEFAULT_FILE_STORAGE =  "storages.backends.sftpstorage.SFTPStorage"
    SFTP_STORAGE_HOST="cdn.fake.org"
    SFTP_STORAGE_ROOT="/var/www/html/media/"
    SFTP_STORAGE_PARAMS = {
      "username": "fake",
      "password": "fake",
      "allow_agent": False,
      "look_for_keys": False,
    }
    MEDIA_ROOT = '/var/www/html/media/'
    MEDIA_URL = 'http://cdn.fake.org/media/'
    STATICFILES_STORAGE = "django.contrib.staticfiles.storage.StaticFilesStorage"
    STATIC_URL = '/static/'
    STATIC_ROOT = os.path.join(BASE_DIR, 'public', 'static')
    STATICFILES_DIRS = (
        os.path.join(BASE_DIR, 'backend', 'static'),
    )

those are my settings on prod, I have update them as @Re said

fringe briar
#

And It doesn't works? Do you already try to read the django-storages documentation?

gleaming shore
#

yes... its not working

#

yes I have read the doc. of django-storages[sftp] I have do the same conf. with no luck

#

I was saying about local to insure that the issue is not from the sftp ...

hybrid widget
#

You said you had a crash? Please post the details.

fringe briar
#

Ok so... what is the error code the browsear is giving you? You have that at least?

gleaming shore
#

Failed to load resource: the server responded with a status of 404 (Not Found)
!

#

and the image link is from the media on my website ...

#

so its normal that it shows 404

fringe briar
#

And the server is dont giving you any logs when you do that?

gleaming shore
#

what I mean, it looks for the image in the default media not on the SFTP server media

fringe briar
#

Ok so the backend is not written the file on the sftp server ?

#

When is in prod

gleaming shore
#

yes exactly

#

its not even communicate with the SFTP

#

because the MEDIA_URL is reading from the folder media in my website server not from the standalone sftp

#

so the issue definitly in the settings

fringe briar
hybrid widget
hybrid widget
gleaming shore
#

okay @boxed, hmm lets say the application doesn't understand the settings that I have put and also not showing big errors .. hhh, πŸ™Œ
yes boxed I have nginx path mapping for media.

fringe briar
#

base_url or SFTP_BASE_URL

Default: Django MEDIA_URL setting

The URL to serve files from.

Is storages looking for SFTP_BASE_URL instead of MEDIA_URL?

gleaming shore
#

yes it seems from the doc. that I should provide base_url. I will update it and see

fringe briar
#

Ok gl

gleaming shore
#

@hybrid widget this is my nginx conf.

server {
    server_name  test.fake.org;
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/fake.org/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/fake.org/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
    
    client_max_body_size 500M;
    location = /favicon.ico { access_log off; log_not_found off; }

    # gzip on;
    # gzip_comp_level 2;
    # gzip_types      text/plain text/css application/javascript text/xml application/xml application/xml+rss image/svg+xml;

    gzip  on;
    gzip_vary on;
    gzip_min_length 10240;
    gzip_proxied expired no-cache no-store private auth;
    gzip_types text/plain text/css text/xml application/javascript application/xml application/xml+rss image/svg+xml;
    gzip_disable "MSIE [1-6]\.";

    
    location / {
        proxy_pass http://unix:/run/fake/socket;
        include     proxy_params;          
    }

    location /static/ {
        alias /var/www/fake_website/public/static/;
        access_log off;
        log_not_found off;
        expires max;
    }

    location /media/ {
        alias /var/www/html/media/;
        access_log off;
        log_not_found off;
        expires max;
    }
    
}

alias of /media/ : /var/www/html/media/ : this path is on sftp server, is that true ?

gleaming shore
# fringe briar base_url or SFTP_BASE_URL Default: Django MEDIA_URL setting The URL to serve ...

I have update it now: it looks like this

USE_NNUH = os.environ.get('USE_NNUH') == 'TRUE'
if USE_NNUH:
    # Static files settings
    DEFAULT_FILE_STORAGE = "storages.backends.sftpstorage.SFTPStorage"
    SFTP_STORAGE_HOST= "cdn.fake.org"
    SFTP_STORAGE_ROOT= "/var/www/html/media/"
    SFTP_STORAGE_PARAMS = {
      "username": "fake",
      "password": "fake",
      "allow_agent": False,
      "look_for_keys": False,
    }
    SFTP_BASE_URL = "http://cdn.fake.org/media/"

is this okay ?

fringe briar
gleaming shore
#

no.. the same, the image after uploading say 404 and its url are from the normal media not from cdn.fake.org

hybrid widget
gleaming shore
#

so the sftp working pretty well, but the django website not even reach the sftp

fading wadi
#

USE_NNUH = os.environ.get('USE_NNUH') == 'TRUE' if USE_NNUH:

Are you sure this turns out to be true? Because if it doesn't have the url you've specificed, then it sounds like it's not using the backend you specified

#

I didnt read all of the text

gleaming shore
#

USE_NNUH=TRUE ( put in the .env file on server).. and I have added print statements inside the "if-statment" and its printed

fading wadi
#

I'd suggest showing the entire settings file then and cut out sensitive parts ^^

#

Just to make sure we're all on the same page

gleaming shore
#

okay no problem. I will show it

from .base import *
DEBUG = True
ALLOWED_HOSTS = ['test.fake.org']
DATABASES = {
    'default': {
        'CONN_MAX_AGE': 0,
        'ENGINE': 'django.db.backends.mysql',
        'HOST': 'localhost',
        'NAME': '',
        'PASSWORD': '',
        'PORT': '',
        'USER': ''
    }
}
DEFAULT_AUTO_FIELD = 'django.db.models.AutoField'
FILER_ENABLE_PERMISSIONS = True
FILER_ALLOW_REGULAR_USERS_TO_ADD_ROOT_FOLDERS = False
FILE_UPLOAD_PERMISSIONS = 0o644
FILE_UPLOAD_MAX_MEMORY_SIZE = 8388608
FILER_SUBJECT_LOCATION_IMAGE_DEBUG = True

SECURE_SSL_REDIRECT= False
# Email settings
EMAIL_BACKEND = "django.core.mail.backends.smtp.EmailBackend"
EMAIL_HOST = 'smtp.gmail.com'
EMAIL_PORT = 465
EMAIL_HOST_USER = os.environ.get('EMAIL_HOST_USER')
EMAIL_HOST_PASSWORD = os.environ.get('EMAIL_HOST_PASSWORD')
EMAIL_USE_SSL = True
EMAIL_USE_TLS = False


# sftp
USE_NNUH = os.environ.get('USE_NNUH') == 'TRUE'
if USE_NNUH:
    print("111111")
    # Static files settings
    DEFAULT_FILE_STORAGE = "storages.backends.sftpstorage.SFTPStorage"
    SFTP_STORAGE_HOST= "cdn.fake.org"
    SFTP_STORAGE_ROOT= "/var/www/html/media/"
    SFTP_STORAGE_PARAMS = {
      "username": "fake",
      "password": "fake",
      "allow_agent": False,
      "look_for_keys": False,
    }
    SFTP_BASE_URL = "http://cdn.fake.org/media/"
else:
    # Local development settings
    STATIC_URL = '/static/'
    STATIC_ROOT = os.path.join(BASE_DIR, 'public', 'static')
    STATICFILES_DIRS = (
        os.path.join(BASE_DIR, 'backend', 'static'),
    )
    MEDIA_URL = '/media/'
    MEDIA_ROOT = os.path.join(BASE_DIR, 'public', 'media')
    DEFAULT_FILE_STORAGE = 'django.core.files.storage.FileSystemStorage'
    STATICFILES_STORAGE = "django.contrib.staticfiles.storage.StaticFilesStorage"
gleaming shore
#

I feel hopeless, It seems that the only solution is to do upgrade for the django version πŸ˜” and this is big big story, because I have many dependencies

hybrid widget
#

I don't think that will help honestly

#

Although I do also think it's not going to be as bad as you think. Django doesn't really change that much over time for most users.

gleaming shore
#

so u mean I could give it a try ,, to upgrade ?

hybrid widget
#

I mean.. you could. But I don't think it will make a difference. If you want the same version you can downgrade your local copy.

#

The files are on disk, and you have an nginx mapping. This should be working, even if you shut down django totally.

gleaming shore
#

okay, what I have not mention on the post, that the production website is django cms πŸ₯²
and the local website is django
so I think that django cms MAYBE require another configurations

gleaming shore
hybrid widget
#

The ones you can successully upload?

gleaming shore
#

yes on the sftp I successfully have folder created by the local env.

hybrid widget
#

"by the local env"? I don't understand that

gleaming shore
#

I mean by upload from the local env.

hybrid widget
#

yea ok. So you have files on disk. And nginx is set up to serve those files.

#

Right?

gleaming shore
#

nginx on sftp or on the website ?

hybrid widget
#

hmm. that question seems strange. How is nginx related to the sftp server at all?

gleaming shore
#

sftp server created by installing sftp on nginx server

hybrid widget
#

Hmm, ok I see. I think part of the problem here is communication. You are using terms subtly incorrectly.

You can't install anything on an nginx server, because that makes no sense. You can install something on the SAME server as you are also running ngxin on. "Nginx server" refers to the process running on a physical machine (also, unfortunately, called a server).

gleaming shore
#

sftp on IP and the website is on different IP
but I can connect with the sftp from the website shell
sftp username@.....

hybrid widget
#

So you have a hardware server running an sftp server, an nginx server, and a gunicorn/django server.

#

Or maybe you have multiple servers, it's not at all clear!

gleaming shore
#

hardware ?

hybrid widget
#

One computer == one hardware server.

green urchin
#

Nginx can serve SFTP now? πŸ˜…

gleaming shore
#

Im not who do it hhhh

green urchin
#

You're probably using vsftp or something

gleaming shore
#

I have given an IP and they said you could upload to this ip

#

and I have tried uploading to it from my local django website and it upload and serve the files correctly

#

so what suppose this to be ?!?

#

sftp is configured on port 22 on this IP

#

and also maybe they make nginx in order I could browse the files by browser

#

cheap cdn

#

hahaha

#

πŸ˜‚πŸ˜†

hybrid widget
#

I don't understand anything about what the setup is, how many machines there are, or what the problem even is at this point. :(

fringe briar
#

The only thing that changes if you update de versiΓ³n is justo de way on how you declares the storages, so i donk think i would works either

gleaming shore
gleaming shore
hybrid widget
#
  1. how many actual machines are we talking about here? Virtual or hardware. A "machine" here is a computer that you can (hopefully) SSH into.
gleaming shore
#

maybe I dont understand ur questions, thats why I have not answer

hybrid widget
#
  1. What is the problem again? I don't even know! Is the URL wrong? In that case, what is the url now, and what should it be?
hybrid widget
#

(Although if you don't understand, and don't understand that you don't understand, yea that's going to be hard)

gleaming shore
gleaming shore
#

until now, do I answer ur questions ?

hybrid widget
hybrid widget
#

The first question is answered πŸ‘

gleaming shore
hybrid widget
#

(I thought before that the sftp server, nginx, and django all ran on the same machine)

gleaming shore
#

Nooooo !! sure they are on different machine

hybrid widget
gleaming shore
hybrid widget
hybrid widget
gleaming shore
gleaming shore
#

so I confirmed that my website is not even write to the sftp nor to my local storage inside the website folder in nginx

#

is that answer your questions ?

#

I have explained every single thing !