#Best way to store http.Request for longer times

38 messages · Page 1 of 1 (latest)

mild lynx
#

What specifically is ineffective about it? It sounds like a good solution to me.

#

Is it performance? Memory usage? Both? Are you storing the serialised requests in-process or elsewhere? Why not just keep them as *http.Request?

fringe tiger
#

PS. This may sound like a XY problem
indeed!
but is not one
are you sure 🙂 ?

if you're going to batch things, why not fully process the incoming requests at least up to the point of extracting what you want to batch for instance?

mild lynx
#

What are you using sync.Pool for? I'm assuming the byte buffers?

#

Have you profiled it at all?

#

Sounds like it could benefit from some horizontal scaling, at the very least

#

When you say "store them", what do you mean by that?

mild lynx
#

That's not really what sync pool is designed for. It's more for things like buffers that you're going to reuse a bunch, and then put back. Saves on allocations, but you wouldn't see any benefit from just throwing all your requests in there

#

I don't even think it would be possible, if you need to look up a specific request by id or something

mild lynx
#

Are you sure the DB isn't a huge performance drain? A lot of round-trips can quickly add up and slow down your entire application.

#

or even the db itself may not be capable of handling the level of traffic you're talking about

#

Can you share some code? Particularly around your use of http.Request.Write and http.ReadRequest

#

I don't think you can get around having to create hundreds of *http.Requests per second. That's just part of the problem. What about encoding the request in a different, intermediary format. Does it need to be a *http.Request specifically? Or could you pull out the information you're interested in?

#

If you can do that, you could even use Protobuf for encoding and decoding the payloads into the database.

#

I think part of the performance hit is having to parse the whole http request every time. But if you've done it once, you don't really need to do it again.

mild lynx
#

Say for instance you only care about the URL and the body, you could have your own representation of a http request like

type Request struct {
    URL  string
    Body []byte
}
#

Or even

syntax = "proto3";

message Request {
  string url = 1;
  bytes body = 2;
}
#

That way you're only serialising/deserialising what you need, you're not parsing the request, and re-encoding it perfectly every single time.

mild lynx
#

I would strongly urge you to profile it, if you haven't already

#

You might find that the request encoding/decoding isn't your problem.

civic geode
#

One thing I noticed about http.ReadRequest is that it requires a bufio.Reader which is great if you're reading from a network connection. If you're loading a request from the database already into []byte and then creating a bytes.NewReader on top of it, there is little need to have a large bufio.Reader on top of it. The default buffer size for bufio.Reader is 4KB. In my code I reduced it to 64b and that made pprof much happier.

    httpReq, err := http.ReadRequest(bufio.NewReaderSize(bytes.NewReader(data), 64))
fringe tiger
#

not sure what you mean @civic geode , just use a bytes.Reader

civic geode
#

I believe that http.ReadRequest mandates a bufio.Reader which is a *struct. You cannot use a bytes.Reader as is. Gotta wrap it.

#

bufio.Reader allocates an internal buffer of 4K by default

fringe tiger
civic geode
#

Maybe you’re confusing NewRequest with ReadRequest?

#

I thought I read above that ReadRequest was being used

fringe tiger
#

if you stored the content of a request into a db or otherwise have the bytes, you don’t need anything but a bytes.Reader to send said request

#

but looks like you’re talking about incoming request processing

#

for which you’d use io.ReadAll or io.Copy ?

#

(but you’re right they mentioned http.ReadRequest and I am not sure why one would use that (starting with not working with h2))

civic geode
#

My understanding was that s/he’s reading the bytes data of the request from the database into http.Request, manipulating it, then re-persisting it as bytes back to the database.

#

If my assumption is correct: I tested http reading and writing at over 10,000 times per second. I think the problem lies elsewhere.

fringe tiger
#

I can do 500k http/sec on a big enough machine with fortio above (no DB involved though but the server side is normal http reading/echo using just io.Copy())

#

30k/sec on my laptop

civic geode
#

I’m not surprised. The 10,000 number was with no concurrency and with network latency to/from the messaging bus. My point is that if they’re having challenge with a few 100/s, the problem is likely elsewhere.