#django-storages with sftp, django v3.2
1 messages Β· Page 1 of 1 (latest)
you mean by command line ?
The method doesn't matter. I mean, can you connect to the sftp server and upload files without any error message?
yes I can connect to it and upload files by those settings on my local environemt.
hmm.. well that's strange. Can you see those files on the file system on the server?
yes, but when I add the same settings on LIVE, it crashes, I will attach a screenshot of whats happening on production
I remember having a custom paramiko based server at a previous job, and that was tons of super custom code. How do you do it?
okay on local I have those settings:
USE_NNUH = config('USE_NNUH') == 'TRUE'
if USE_NNUH:
# Static files settings
STORAGES = {
"default": {
"BACKEND": "storages.backends.sftpstorage.SFTPStorage",
"OPTIONS": {
"host": "cdn.fake.org",
"root_path": "/var/www/html/media/",
"params": {'username': 'fake', 'password': 'fake', 'allow_agent': False},
},
},
"staticfiles": {
"BACKEND": "django.contrib.staticfiles.storage.StaticFilesStorage",
}
}
MEDIA_ROOT = '/var/www/html/media/'
MEDIA_URL = 'http://cdn.fake.org/media/'
else:
# Local development settings
STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(PROJ_DIR, 'static_collected/')
STATICFILES_DIRS = [BASE_DIR / 'static', ]
MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(PROJ_DIR, 'media/')
STORAGES = {
"default": {
"BACKEND": "django.core.files.storage.FileSystemStorage",
},
"staticfiles": {
"BACKEND": "django.contrib.staticfiles.storage.StaticFilesStorage",
}
}
those previous settings are work 100% correctly, using django-storages[sftp], django 4.2 , paramiko, bcrypt==3.2.0, cryptography==41.0.7
I'm able to upload and even read the files on django admin side: when I click any image ( that I have uploaded) it open in fake.nnuh.org, and I even see the image in the sftp media folder
on PROD I have django 3.2 :/
A crash seems like the first thing to look at... do you have sentry set up so you can get the full crash logs and stuff?
no actually I have not setup sentry on PROD, I'm relying on what debug=True shows
You should always have debug false in prod, as you could leak sensitive information to hackers otherwise.
wait.. but you said at the start "it doesn't write the file to the sftp server"?
I don't understand the problem anymore. You can upload, you can view the files... what was the problem?
okay: what I have posted yesterday is on PROD env.
and I'm saying to you that the same settings work on local env. !!
everything works correctly on local, but on prod ( its a production but not public until now) so no worries if I make DEBUG=True...
Do u understand me now ?
You have django 3.2 right?
yes, on PROD, and I cannot do upgrade due to many reasons... do u think the version is a problem ?
So the problem is that your backend is not even connecting with your object storage right?
yah and it appear like it doesn't see the sftp at all
I think it can be because this:
STORAGES = {
"default": {
"BACKEND": "storages.backends.sftpstorage.SFTPStorage",
"OPTIONS": {
"host": "cdn.fake.org",
"root_path": "/var/www/html/media/",
"params": {'username': 'fake', 'password': 'fake', 'allow_agent': False},
},
},
"staticfiles": {
"BACKEND": "django.contrib.staticfiles.storage.StaticFilesStorage",
}
}
since this is the way that you declares the storages on django 4.2 or superior, for <4.2 you need to do it in this way:
DEFAULT_FILE_STORAGE = "storages.backends.s3.S3Storage"
STATICFILES_STORAGE = "storages.backends.s3.S3Storage"
okay I will try to define them as u said and return back with the result
I know.
If you have a crash, you should post the traceback and error message. It almost always contains a ton of very useful information for figuring out the problem.
I noticed also that Re had to guess what the problem was, and you answered yes. It would have been much better had you stated the problem.
Also you started by saying "it doesn't write the file to the sftp server" which then later was not true. Was that because you managed to fix something in the meantime and didn't communicate to us?
Getting help is about communicating clearly.
That's what I dont understand, there is an error or not?
So it was stated here π€·ββοΈ
But it was also stated that files are not written to the server, and that files ARE written to the server.
Yeah but if it is written files, the problem it might to be because the storages but on that message @gleaming shore is saying the app crashing but, if it crash what is the output of the error?
okay:
- I'm sorry If I make any misunderstanding to you guys
- the post ( originally) talk about my problem on production which is based on django v3.2
- what I print today is the settings that I have them locally for django v4, I have tried the settings locally and upload an image and I'm able to view the image successfully from the url of the sftp server, and I have check the sftp server, it has the image that I have uploaded ------ LOCAL THIS IS LOCAL
- lets talk about PROD -_-, on production I have change the conf to make it suitable for django 3.2 but still when I upload image it shows like corrupted image and the link of the image ( from inspect element) is domained on the website itself not from cdn. ( that's what I mean by crashes, I don't see any error/big error)
USE_NNUH = os.environ.get('USE_NNUH') == 'TRUE'
if USE_NNUH:
# Static files settings
DEFAULT_FILE_STORAGE = "storages.backends.sftpstorage.SFTPStorage"
SFTP_STORAGE_HOST="cdn.fake.org"
SFTP_STORAGE_ROOT="/var/www/html/media/"
SFTP_STORAGE_PARAMS = {
"username": "fake",
"password": "fake",
"allow_agent": False,
"look_for_keys": False,
}
MEDIA_ROOT = '/var/www/html/media/'
MEDIA_URL = 'http://cdn.fake.org/media/'
STATICFILES_STORAGE = "django.contrib.staticfiles.storage.StaticFilesStorage"
STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, 'public', 'static')
STATICFILES_DIRS = (
os.path.join(BASE_DIR, 'backend', 'static'),
)
those are my settings on prod, I have update them as @Re said
And It doesn't works? Do you already try to read the django-storages documentation?
yes... its not working
yes I have read the doc. of django-storages[sftp] I have do the same conf. with no luck
I was saying about local to insure that the issue is not from the sftp ...
You said you had a crash? Please post the details.
Ok so... what is the error code the browsear is giving you? You have that at least?
Failed to load resource: the server responded with a status of 404 (Not Found)
!
and the image link is from the media on my website ...
so its normal that it shows 404
And the server is dont giving you any logs when you do that?
https://test.nnuh.org/media/filer_public_thumbnails/filer_public/26/2c/262ce88e-4120-4201-823d-6bf5decae398/picture20.png__80x80_crop_subsampling-2.png this is the generated link of the image, which gives 404
what I mean, it looks for the image in the default media not on the SFTP server media
yes exactly
its not even communicate with the SFTP
because the MEDIA_URL is reading from the folder media in my website server not from the standalone sftp
so the issue definitly in the settings
https://django-storages.readthedocs.io/en/latest/backends/sftp.html
Look, ok the change I mention II good to start but leet me read some more so I maybe have an idea of this
Just fyi, that's not considered a "crash"
Do you have an nginx path mapping set up for /media/?
okay @boxed, hmm lets say the application doesn't understand the settings that I have put and also not showing big errors .. hhh, π
yes boxed I have nginx path mapping for media.
base_url or SFTP_BASE_URL
Default: Django MEDIA_URL setting
The URL to serve files from.
Is storages looking for SFTP_BASE_URL instead of MEDIA_URL?
yes it seems from the doc. that I should provide base_url. I will update it and see
Ok gl
@hybrid widget this is my nginx conf.
server {
server_name test.fake.org;
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/fake.org/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/fake.org/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
client_max_body_size 500M;
location = /favicon.ico { access_log off; log_not_found off; }
# gzip on;
# gzip_comp_level 2;
# gzip_types text/plain text/css application/javascript text/xml application/xml application/xml+rss image/svg+xml;
gzip on;
gzip_vary on;
gzip_min_length 10240;
gzip_proxied expired no-cache no-store private auth;
gzip_types text/plain text/css text/xml application/javascript application/xml application/xml+rss image/svg+xml;
gzip_disable "MSIE [1-6]\.";
location / {
proxy_pass http://unix:/run/fake/socket;
include proxy_params;
}
location /static/ {
alias /var/www/fake_website/public/static/;
access_log off;
log_not_found off;
expires max;
}
location /media/ {
alias /var/www/html/media/;
access_log off;
log_not_found off;
expires max;
}
}
alias of /media/ : /var/www/html/media/ : this path is on sftp server, is that true ?
I have update it now: it looks like this
USE_NNUH = os.environ.get('USE_NNUH') == 'TRUE'
if USE_NNUH:
# Static files settings
DEFAULT_FILE_STORAGE = "storages.backends.sftpstorage.SFTPStorage"
SFTP_STORAGE_HOST= "cdn.fake.org"
SFTP_STORAGE_ROOT= "/var/www/html/media/"
SFTP_STORAGE_PARAMS = {
"username": "fake",
"password": "fake",
"allow_agent": False,
"look_for_keys": False,
}
SFTP_BASE_URL = "http://cdn.fake.org/media/"
is this okay ?
It looks like thats what the documentation have
no.. the same, the image after uploading say 404 and its url are from the normal media not from cdn.fake.org
hmm, that looks ok to me. The files are in /var/www/html/media/ right? Like, you've verified that?
boxed: I have try upload video by drag and drop into the sftp. and I have successfully open it in the sftp browser: http://cdn.fake.org/media/videos_uploaded/clinics.mp4
so the sftp working pretty well, but the django website not even reach the sftp
USE_NNUH = os.environ.get('USE_NNUH') == 'TRUE' if USE_NNUH:
Are you sure this turns out to be true? Because if it doesn't have the url you've specificed, then it sounds like it's not using the backend you specified
I didnt read all of the text
USE_NNUH=TRUE ( put in the .env file on server).. and I have added print statements inside the "if-statment" and its printed
I'd suggest showing the entire settings file then and cut out sensitive parts ^^
Just to make sure we're all on the same page
okay no problem. I will show it
from .base import *
DEBUG = True
ALLOWED_HOSTS = ['test.fake.org']
DATABASES = {
'default': {
'CONN_MAX_AGE': 0,
'ENGINE': 'django.db.backends.mysql',
'HOST': 'localhost',
'NAME': '',
'PASSWORD': '',
'PORT': '',
'USER': ''
}
}
DEFAULT_AUTO_FIELD = 'django.db.models.AutoField'
FILER_ENABLE_PERMISSIONS = True
FILER_ALLOW_REGULAR_USERS_TO_ADD_ROOT_FOLDERS = False
FILE_UPLOAD_PERMISSIONS = 0o644
FILE_UPLOAD_MAX_MEMORY_SIZE = 8388608
FILER_SUBJECT_LOCATION_IMAGE_DEBUG = True
SECURE_SSL_REDIRECT= False
# Email settings
EMAIL_BACKEND = "django.core.mail.backends.smtp.EmailBackend"
EMAIL_HOST = 'smtp.gmail.com'
EMAIL_PORT = 465
EMAIL_HOST_USER = os.environ.get('EMAIL_HOST_USER')
EMAIL_HOST_PASSWORD = os.environ.get('EMAIL_HOST_PASSWORD')
EMAIL_USE_SSL = True
EMAIL_USE_TLS = False
# sftp
USE_NNUH = os.environ.get('USE_NNUH') == 'TRUE'
if USE_NNUH:
print("111111")
# Static files settings
DEFAULT_FILE_STORAGE = "storages.backends.sftpstorage.SFTPStorage"
SFTP_STORAGE_HOST= "cdn.fake.org"
SFTP_STORAGE_ROOT= "/var/www/html/media/"
SFTP_STORAGE_PARAMS = {
"username": "fake",
"password": "fake",
"allow_agent": False,
"look_for_keys": False,
}
SFTP_BASE_URL = "http://cdn.fake.org/media/"
else:
# Local development settings
STATIC_URL = '/static/'
STATIC_ROOT = os.path.join(BASE_DIR, 'public', 'static')
STATICFILES_DIRS = (
os.path.join(BASE_DIR, 'backend', 'static'),
)
MEDIA_URL = '/media/'
MEDIA_ROOT = os.path.join(BASE_DIR, 'public', 'media')
DEFAULT_FILE_STORAGE = 'django.core.files.storage.FileSystemStorage'
STATICFILES_STORAGE = "django.contrib.staticfiles.storage.StaticFilesStorage"
I feel hopeless, It seems that the only solution is to do upgrade for the django version π and this is big big story, because I have many dependencies
I don't think that will help honestly
Although I do also think it's not going to be as bad as you think. Django doesn't really change that much over time for most users.
so u mean I could give it a try ,, to upgrade ?
I mean.. you could. But I don't think it will make a difference. If you want the same version you can downgrade your local copy.
The files are on disk, and you have an nginx mapping. This should be working, even if you shut down django totally.
okay, what I have not mention on the post, that the production website is django cms π₯²
and the local website is django
so I think that django cms MAYBE require another configurations
how ? I dont understand you correctly here
You have a folder on the server that contains media files right?
The ones you can successully upload?
yes on the sftp I successfully have folder created by the local env.
"by the local env"? I don't understand that
I mean by upload from the local env.
yea ok. So you have files on disk. And nginx is set up to serve those files.
Right?
nginx on sftp or on the website ?
hmm. that question seems strange. How is nginx related to the sftp server at all?
sftp server created by installing sftp on nginx server
Hmm, ok I see. I think part of the problem here is communication. You are using terms subtly incorrectly.
You can't install anything on an nginx server, because that makes no sense. You can install something on the SAME server as you are also running ngxin on. "Nginx server" refers to the process running on a physical machine (also, unfortunately, called a server).
sftp on IP and the website is on different IP
but I can connect with the sftp from the website shell
sftp username@.....
So you have a hardware server running an sftp server, an nginx server, and a gunicorn/django server.
Or maybe you have multiple servers, it's not at all clear!
hardware ?
One computer == one hardware server.
Nginx can serve SFTP now? π
Im not who do it hhhh
You're probably using vsftp or something
I have given an IP and they said you could upload to this ip
and I have tried uploading to it from my local django website and it upload and serve the files correctly
so what suppose this to be ?!?
sftp is configured on port 22 on this IP
and also maybe they make nginx in order I could browse the files by browser
cheap cdn
hahaha
ππ
I don't understand anything about what the setup is, how many machines there are, or what the problem even is at this point. :(
The only thing that changes if you update de versiΓ³n is justo de way on how you declares the storages, so i donk think i would works either
search google on how to create cdn with nginx, and you will understand what I'm saying
yes I think the same, but I must find a solution for this problem, actually the project is django cms and I have ask the same question on django cms community, hopefully someone maybe has idea about the same setup
No that's not the problem. The problem is about communication and words. You have said self-conflicting things, been unclear about how many machines are involved, confused the two meanings of "server", not answered questions asked, and when we say we are confused you don't seem to react and try to clear up the confusion.
- how many actual machines are we talking about here? Virtual or hardware. A "machine" here is a computer that you can (hopefully) SSH into.
maybe I dont understand ur questions, thats why I have not answer
- What is the problem again? I don't even know! Is the URL wrong? In that case, what is the url now, and what should it be?
If you don't understand the question it is enormously important to say so.
(Although if you don't understand, and don't understand that you don't understand, yea that's going to be hard)
- we have 2 IPs, one dedicated for the django website and another IP for the sftp server ( 2 virtual machine)
- the problem is : the conf. that I have done not making my website write the files to the sftp server
until now, do I answer ur questions ?
IPs are not the same as machines btw, a machine can have multiple IPs. In fact they ALWAYS do!
No, that made no sense. The files are written to disk via sftp ONLY right?
The first question is answered π
now this: I dont understand here by ONLY ?!?
(I thought before that the sftp server, nginx, and django all ran on the same machine)
Nooooo !! sure they are on different machine
Files are uploaded via sftp. You never upload files via the browser. Correct?
another Nooo, I have upload files by website, but they are not stored in the sftp
- hmm.. nginx runs on the same machine as django right?
Ok, so you can upload files in two ways and you want them to be stored in the same place?
yes its django website with nginx server
I have upload files to the website, but they appear as corrupted and when I make inspect element over those images I have seen that the url ( src=) link are on my website ...... and those links gives 404
so I confirmed that my website is not even write to the sftp nor to my local storage inside the website folder in nginx
is that answer your questions ?
I have explained every single thing !