Fetching compressed objects from origin

[vagrant sundial]

I've got a bunch of compressed objects in the origin (GCS bucket) and if I hit them directly (with curl using --compressed), they are returned to me gzipped.

Now, if I hit my worker with --compressed and it goes and fetches the same object from the origin, it comes back uncompressed. I remember reading somewhere that regardless of what headers are sent in the request to Cloudflare, it will always attempt to fetch from the origin using compression and then depending on the user request, it will return compressed or uncompressed, but that doesn't seem to be the case.

Any ideas? TIA.

[vagrant sundial]

BTW my current workaround is to send request.headers along when calling fetch() but I wonder why that's required.

[stone igloo]

Sounds like standard Accept-Encoding

If you don't indicate GZIP support with Accept-Encoding then it would be decompressed on-the-fly

[vagrant sundial]

@stone igloo ok so sending request headers to the origin sounds reasonable?

[stone igloo]

If you want the client's compression support to be respected then yeah

[vagrant sundial]

Thanks!

Some of the files are highly compressed (10x) so yeah we want to send them compressed if the client requests it

[stone igloo]

The part you found about 'it will always attempt to fetch from the origin using compression and then depending on the user request, it will return compressed or uncompressed' holds true

Your Worker is the 'user' in this scenario though

You might think it is equivalent of Cloudflare talking to the origin but it's actually equivalent of a user talking to Cloudflare

[vagrant sundial]

Right yeah

So it probably came back to CF compressed but due to the lack of accept encoding, it got uncompressed on-the-fly

[stone igloo]

https://kian.org.uk/cloudflares-transparent-decompression-how-does-it-work/ explains the behaviour of a client's Accept-Encoding but the tl;dr is Cloudflare requests compressed content and will decompress it on-the-fly back to the user if required

[stone igloo]

Yup

It's so that even if you don't support compression as a client, you save on bandwidth costs at your origin as it would be requested compressed.

[vagrant sundial]

I guess CF doesn't compress on the fly then? Seeing as how CF requests the content without accept encoding so it got uncompressed, given that the original request had accept encoding gzip, it doesn't then compress it before sending to the end user?

[stone igloo]

It will compress on-the-fly

https://developers.cloudflare.com/support/speed/optimization-file-size/what-will-cloudflare-compress/

[vagrant sundial]

@stone igloo so hrm why isn't the content returned by fetch() getting compressed on the fly? 🤔

[stone igloo]

Sounds like an issue with your specific scenario/origin

[vagrant sundial]

the origin URL is:

https://storage.googleapis.com/about.gitlab-review.app/1006-qa-fix-color-on-links-for-campus-page/index.html

This is what fetch() is getting... if you hit that with curl, it comes back gzipped. Not sure what else I'm doing wrong for this content not to get compressed on the fly... the content-type is set to text/html so it should get compressed?

Cache-Control: public, max-age=60,no-transform

hrm the doc does say to set cache-control to no-transform to avoid compression 🤔

yep that did it.. removed no-transform and it gzipped fine 😄