#Rust HTTP Request Returns Garbage, Works in Python?

12 messages · Page 1 of 1 (latest)

frozen current
#

Hi all, I'm trying to create a HTTP request to the Minecraft server webpage to get the latest version of Minecraft as a string.

I wrote some pretty simple code:

#[tokio::main]
async fn request_thread() {
  println!("Starting request");

  let mut headers = HeaderMap::new();
  headers.insert(
    "Accept",
    HeaderValue::from_static(
      "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8",
    ),
  );
  headers.insert(
    "Accept-Language",
    HeaderValue::from_static("en-GB,en;q=0.5"),
  );
  headers.insert(
    "Accept-Encoding",
    HeaderValue::from_static("gzip, deflate, br"),
  );
  headers.insert("DNT", HeaderValue::from_static("1"));
  headers.insert("Connection", HeaderValue::from_static("keep-alive"));
  headers.insert("Upgrade-Insecure-Requests", HeaderValue::from_static("1"));
  headers.insert("Sec-Fetch-Dest", HeaderValue::from_static("document"));
  headers.insert("Sec-Fetch-Mode", HeaderValue::from_static("navigate"));
  headers.insert("Sec-Fetch-Site", HeaderValue::from_static("cross-site"));
  headers.insert("TE", HeaderValue::from_static("trailers"));
  headers.insert(
    "User-Agent",
    HeaderValue::from_static(
      "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:108.0) Gecko/20100101 Firefox/108.0",
    ),
  );

  let client = reqwest::Client::builder()
    .default_headers(headers)
    .build()
    .unwrap();

  let resp = client
    .get("https://www.minecraft.net/en-us/download/server")
    .send()
    .await
    .unwrap()
    .text()
    .await
    .unwrap();

  println!("Request ended: {}", resp);
}

I then run this elsewhere in my program like this:

std::thread::spawn(request_thread);

However, instead of the expected HTML file, it returns complete garbage (see image #1)

Doing the exact same request in Python (same URL, same headers) returns the correct HTML file (see image #2)

Surely something's wrong with Rust here, if it works in Python? Am I doing something wrong?

image.png image.png
#

I've tried printing the response to a HTML file, as well as a ton of other things (like trying all the methods text, text_with_charset, or calling bytes() then using str::from_utf8 on the bytes but nothing worked. It seems that reqwest itself is receiving garbage data from this HTTP request. I have no idea what it could be since it works in Python.

#

Anyone with any ideas?

slender bear
frozen current
#

Damn

#

That was a good guess, it worked!

#

I didn't even realise I forgot to put text/html in there

#

But what really confused me is that it got the correct thing in Python

#

So Python must have received gzip ... and unzipped it ..?

#

Thank you for your help 🙂

hollow roost