Trying to upload in chunks, blocked by CORS | Django | Page 1

fast nebula Aug 12, 2024, 8:47 AM

#

Hi,
For my app I'm trying to upload a videofile in chunks. Main reason is to get around the DigitalOcean 100s timeout for uploads as well as the 500MB nGINX file limit, both of wich you can't change on App Platform.
I have set up a view that allows the upload, I have a view that requests a signed URL to upload it to DO Spaces. I also have the JS needed to get these. However: I get blocked by not having the correct CORS header. In my Spaces settings I have the url for my website with all options allowd (GET, PUT, DLETE, POST). Why is it still getting blocked?
See below for code:

#

📎 HTML_TEMPLATE.txt

#

and the views:

class UploadView(LoginRequiredMixin,generic.CreateView):
    template_name = "Academy/upload.html"
    model = BlogVideo
    fields = ['file']
 
    def get_success_url(self):
        return reverse("Academy:upload")
 
    def get_context_data(self, **kwargs):
        context = super(UploadView, self).get_context_data(**kwargs)
        context.update({
            "uploads": BlogVideo.objects.all()
        })
        return context

class SignedURLView(LoginRequiredMixin ,generic.View):
    def post(self, request, *args, **kwargs):
        session = boto3.session.Session()
        client = session.client(
            "s3",
            region_name='ams3',
            endpoint_url=os.environ.get('AWS_ENDPOINT_URL'),
            aws_access_key_id=os.environ.get('AWS_ACCESS_KEY_ID'),
            aws_secret_access_key=os.environ.get('AWS_SECRET_ACCESS_KEY')
        )
 
        url = client.generate_presigned_url(
            ClientMethod="put_object",
            Params={
                "Bucket": "barts-academy",
                "Key": f"media/videos/{json.loads(request.body)['fileName']}",
            },
            ExpiresIn=300,
        )
        return JsonResponse({"url": url})```

stable marten Aug 12, 2024, 8:49 AM

#

Why do you have two domains?

fast nebula Aug 12, 2024, 8:52 AM

#

What do you mean?

#

The error I keep getting is: Access to XMLHttpRequest at 'https://barts-academy.ams3.digitaloceanspaces.com/barts-academy/uploads/bourbon compressed.mp4.part0?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=DO003YQC7UDRTCFP43RY%2F20240812%2Fams3%2Fs3%2Faws4_request&X-Amz-Date=20240812T085139Z&X-Amz-Expires=300&X-Amz-SignedHeaders=host&X-Amz-Signature=a4405358af06c103201a693a11b85933b1a49301859a2a5a37aeaf2aae106234' from origin 'https://bartsacademy.nl' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource.

hollow star Aug 12, 2024, 8:54 AM

#

Sounds like you should ad DO support of the headers arr not set?

#

The error is quite clear.

stable marten Aug 12, 2024, 8:54 AM

#

I mean, why do you use cors in the first place?

#

One project, one domain is the rule of thumb imo

fast nebula Aug 12, 2024, 8:55 AM

#

stable marten I mean, why do you use cors in the first place?

I tried following this: https://justdjango.com/blog/how-to-upload-large-files

Django Tutorial: How to Upload Large Files

This tutorial will show you how to upload large files in a Django project using Digital Ocean Spaces.

fast nebula Aug 12, 2024, 8:55 AM

#

stable marten One project, one domain is the rule of thumb imo

But where do you see the two domains?

fast nebula Aug 12, 2024, 8:55 AM

#

hollow star Sounds like you should ad DO support of the headers arr not set?

I don;t know how, I'm quyite new to this

#

Thanks both for looking into this to help me btw

fast nebula Aug 12, 2024, 8:56 AM

#

stable marten I mean, why do you use cors in the first place?

Because the CORS settings are needed to upload anything to my Spaces bucket, otherwise it gets rejected no?

#

When I set the allowed header to * it works, but thats not safe no?

stable marten Aug 12, 2024, 9:02 AM

#

fast nebula But where do you see the two domains?

CORS means there are two domains. That's all it's for.

stable marten Aug 12, 2024, 9:02 AM

#

fast nebula Because the CORS settings are needed to upload anything to my Spaces bucket, oth...

aha, yea ok, gotcha

fast nebula Aug 12, 2024, 9:03 AM

#

Yes, I have the domain from my website and the DO Spaces bucket I upload all my media and static files to

cedar rover Aug 12, 2024, 9:10 AM

#

I don't use DO Spaces, but googling "digitalocean spaces cors" led me to https://docs.digitalocean.com/products/spaces/how-to/configure-cors/, which may help.

fast nebula Aug 12, 2024, 9:12 AM

#

cedar rover I don't use DO Spaces, but googling "digitalocean spaces cors" led me to <https:...

It didn't. From what I understand I need to set the "Allowed Header" but i do not know what to set it as. The upload works when I set it to *. But that isn't safe.

cedar rover Aug 12, 2024, 9:13 AM

#

Then please share the complete error, as well as the relevant headers from the browser's dev tools network tab.

#

Those are request headers... you need to check the response headers from the spaces server.

#

Also, you should probably not be sharing CSRF Tokens on here 😅

#

(So I deleted that, though note that there are ways to automatically log these)

fast nebula Aug 12, 2024, 9:22 AM

#

Thanks

cedar rover Aug 12, 2024, 9:23 AM

#

Mostly, we care about the access-control-* headers

fast nebula Aug 12, 2024, 9:23 AM

#

The response header:
cache-control: private cf-cache-status: DYNAMIC cf-ray: 8b1f60e2dd6a663e-AMS content-encoding: br content-type: application/json cross-origin-opener-policy: same-origin date: Mon, 12 Aug 2024 09:18:30 GMT referrer-policy: same-origin server: cloudflare vary: Cookie x-content-type-options: nosniff x-do-app-origin: ad3c407b-c8ee-4aef-8962-d6669aa8d3bc x-do-orig-status: 200 x-frame-options: DENY

cedar rover Aug 12, 2024, 9:25 AM

#

They don't seem to be set at all 🤔

fast nebula Aug 12, 2024, 9:27 AM

#

Isn't that what this part of the JS code does?:

const getSignedUrls = async () => {
    let urls = [];
    for (let i = 0; i < totalChunks; i++) {
        const chunkStart = i * chunkSize;
        const chunkEnd = Math.min(chunkStart + chunkSize, file.size);
        const chunk = file.slice(chunkStart, chunkEnd);

        const body = {
            fileName: `${file.name}.part${i}`,
            fileType: file.type
        };

        const response = await fetch("{% url 'Academy:signed-url' %}", {
          method: "POST",
          body: JSON.stringify(body),
          headers: { "Content-Type": "application/json", "X-CSRFToken": "{{ csrf_token }}"}
      });
        console.log('got signed url');
        const { url } = await response.json();
        urls.push(url);
    };
    return urls;```

cedar rover Aug 12, 2024, 9:28 AM

#

No. Headers need to be set on the server, so via the DO Spaces control panel (or whatever it's called).

#

For example, IMDB sets them correctly for api.graphql.imdb.com, as you can see in the dev tools if you open the IMDB landing page.

fast nebula Aug 12, 2024, 9:30 AM

#

I have them set the same way as in this tutorial: https://justdjango.com/blog/how-to-upload-large-files

Django Tutorial: How to Upload Large Files

This tutorial will show you how to upload large files in a Django project using Digital Ocean Spaces.

#

with my url as the origin: bartsacademy.nl

#

But I don;t know what to put as allowed headers

#

As using a * seems to be unsafe?

cedar rover Aug 12, 2024, 9:31 AM

#

Yeah, don't use *.

#

https://*.bartsacademy.nl ?

#

This should work for bartsacademy.nl and www.bartsacademy.nl (and any other subdomain).

fast nebula Aug 12, 2024, 9:36 AM

#

cedar rover This should work for `bartsacademy.nl` and `www.bartsacademy.nl` (and any other ...

No, I have those set separately to prevent using * But I also need to set the ALLOWED HEADERS. The file upload works when I set that to * the file upload works

#

See the last screenshot in the tuutorial to see what I mean

cedar rover Aug 12, 2024, 9:47 AM

#

fast nebula The error I keep getting is: Access to XMLHttpRequest at 'https://barts-academy....

Oh. * is probably fine for this, since the requests should come from your own server, but if you want to just them explicitly, then add all the non-default ones as per the DO Spaces docs.
The tutorial sets x-amz-acl, but I think you may also need to set the ones in this url 👆 ... x-amz-algorithm, etc.

fast nebula Aug 12, 2024, 10:05 AM

#

Thank. Looks like the upload works now.
However: It feels as if the file is now uploaded twice: Once as chunks and then again as the whole file. The video file now shows up twice in my Spaces Bucket. Did I make a mistake in my view?

cedar rover Aug 12, 2024, 10:08 AM

#

I don't see anything glaring, but I also haven't worked with boto3 or DO Spaces, so that I can't help with 😅

fast nebula Aug 12, 2024, 10:09 AM

#

No worries, thanks for your help!

cedar rover Aug 12, 2024, 10:11 AM

#

Nice site though 😄 🥃

fast nebula Aug 12, 2024, 12:43 PM

#

cedar rover Nice site though 😄 🥃

Thanks man!

#

If only I could upload all the videos we made 😦

wispy reef Aug 14, 2024, 2:23 PM

#

fast nebula If only I could upload all the videos we made 😦

Why you cannot upload all your videos?

#

So I would do something like this @fast nebula :

Ensure that your CORS configuration in DigitalOcean Spaces includes the necessary headers. Here's an example configuration that you can use: [ { "AllowedHeaders": ["*"], "AllowedMethods": ["GET", "POST", "PUT", "DELETE", "HEAD"], "AllowedOrigins": ["https://bartsacademy.nl"], "ExposeHeaders": [], "MaxAgeSeconds": 3000 } ]
Ensure that the signed URL generation includes the necessary headers. In your SignedURLView, you might need to adjust the parameters to include the required headers:
` class SignedURLView(LoginRequiredMixin, generic.View):
def post(self, request, *args, **kwargs):
session = boto3.session.Session()
client = session.client(
"s3",
region_name='ams3',
endpoint_url=os.environ.get('AWS_ENDPOINT_URL'),
aws_access_key_id=os.environ.get('AWS_ACCESS_KEY_ID'),
aws_secret_access_key=os.environ.get('AWS_SECRET_ACCESS_KEY')
)
```
    url = client.generate_presigned_url(
        ClientMethod="put_object",
        Params={
            "Bucket": "barts-academy",
            "Key": f"media/videos/{json.loads(request.body)['fileName']}",
            "ACL": "public-read"
        },
        ExpiresIn=300,
        HttpMethod="PUT"
    )
    return JsonResponse({"url": url})`
```

#

When you send the request to the signed URL, ensure you include the necessary headers:
`function uploadChunk() {
if (currentChunkIndex >= totalChunks) {
setIsSubmitting(false);
submitForm();
return;
}

const chunkStart = currentChunkIndex * chunkSize;
const chunkEnd = Math.min(chunkStart + chunkSize, file.size);
const chunk = file.slice(chunkStart, chunkEnd);
const signedUrl = signedUrls[currentChunkIndex];

var ajax = new XMLHttpRequest();
ajax.upload.addEventListener("progress", (event) => progressHandler(event, currentChunkIndex), false);
ajax.addEventListener("load", (event) => completeChunkHandler(event), false);
ajax.addEventListener("error", errorHandler, false);
ajax.addEventListener("abort", abortHandler, false);
ajax.open("PUT", signedUrl);
ajax.setRequestHeader("Content-Type", file.type);
ajax.setRequestHeader("x-amz-acl", "public-read");
ajax.send(chunk);
console.log('signedURL: ' + signedUrl);
}
`

hollow star Aug 14, 2024, 2:32 PM

#

Is this answer from an AI? If so, please disclose that.

#Trying to upload in chunks, blocked by CORS