#Sending *many* http requests with Rust

57 messages · Page 1 of 1 (latest)

stuck tartan
#

Note that you might be getting rate limited by your ISP or similar.

#

What are you testing this against currently?

#

Have you tested it against a locally running server to rule out that it's the network's fault?

stuck tartan
#

So when you decrease the delay you get timeouts - have you tried anything to debug this?

#

Like checking what's sent across the network with wireshark

#

Or cargo flamegraph to see where program time is spent

#

Or tokio-console maybe

#

Or rust-gdb and find out where timeouts are generated

#

Oh and you say that at 20ms all your CPUs are already saturated

#

So if you go below that it's not very surprising that requests will take longer and longer to complete

#

Are you running this in release mode?

#

Also I think your Mutex<i32> could be an AtomicI32

#

Anyway 40RPS is abysmal. How large are these requests and their payloads and response bodies?

stuck tartan
#

I'd start with flamegraph because that CPU usage isn't normal for a request every 20ms i think

#

Do you have an MRE?

stuck tartan
#

there should be a flamegraph.svg

#

you can view it in your favorite browser

#

it's interactive

#

you can click stuff

stuck tartan
#

Oh, is the Mutex you're using from tokio::sync?

stuck tartan
#

You prob wanna use Tokio Mutexes

#

Only Tokio mutexes

#

There's no Tokio arc

#

Yes

#

Use std arc

#

There's no need for an async version of Arc

#

Because it works just fine

#

You need to lock().await.unwrap() now

#

Maybe no unwrap

#

Unsure

#

How's the RPS?

#

Did it change anything

#

I mean

#

Ah okay

#

Is your repo public?

#

Can you send a screenshot of it

#

Hmmmm

#

Can you try switching to rustls

#

$ cargo add reqwest --features rustls or something

#

Idk

#

Multi thread is the default btw

#

Yeah

#

Try again

#

what about 15ms

#

what does the flamegraph say

#

are you sure you're using rustls now? ferrisThonk

#

oh looks like you do need to change the code a bit

#

on the client builder: builder.use_rustls_tls()

#

don't have a client set up rn but you can donate whatever you'd send me to e.g. givewell and paste the receipt here 🙂

#

also

#

-rule 3

#

-jobs

broken boneBOT
#

Advertising and job listings are disallowed. If you'd like to advertise jobs, we recommend the periodic Who's Hiring thread on the subreddit https://rust.reddit.com/.

stuck tartan
#

wait wrong one

#

anyway