Understanding wakers

[edgy sequoia]

Hi there, I'm trying to understand wakers and contexts better to be able to tell what the proper solution to the following issue is: https://github.com/Gelbpunkt/tokio-websockets/issues/92

The TL;DR: In the process of poll_next on a stream, we currently call poll_flush(cx) on the underlying I/O to ensure that replies mandated by the protocol (e.g. ping/pong and answering close frames) are at least attempted to be sent out when queued up. The actual result - ready or pending - is ignored. This appears to cause an issue in a weird edge case described in the issue: The user has a fair mutex around the stream and uses one half for writing to the sink and one for reading from it. The writer however gets stuck at some point unless the call to poll_flush(cx) is removed. I can also resolve the issue by creating a new context with a no-op waker that I pass to poll_flush (and also by cloning the waker and using that). tokio-console also complains that the waker has been dropped (on the send task).

My question now is what exactly is going on here. To my understanding, it should be perfectly fine as-is. The poll_next call will try to flush the sink and if that returns pending, it is queued to be woken up again when the sink is supposed to be flushed again. Proceeding to do something else in the same context should be fine, I believe. What exactly is going on here at a runtime level? I went through all the documentation for waker and context, but I just cannot make any sense of this behaviour.

[nimble cape]

What is the underlying IO that you are using exactly? I see for example that the TcpStream implementation of poll_flushis a no op: https://github.com/tokio-rs/tokio/blob/master/tokio/src/net/tcp/stream.rs#L1358

With that said, my guess is that the writer half only keeps a single slot to store a waker (since the waker has to be stored somewhere) and that calling a second poll_* method on the same object before causes the second waker to overwrite the first one.

[edgy sequoia]

The underlying I/O is usually a TCP stream, but not always. The Websocket stream's poll_flush (the one I was talking about, sorry for not clarifying that) performs I/O via poll_write_buf from tokio-util and then calls poll_flush on e.g. the TCP stream. Anyways, having only a single slot would be really weird, since the documentation for AsyncWrite only says:

the current future’s task is scheduled to get unparked when the object is writable

which I understand as it being added to a list of tasks to get unparked instead of replacing the last one.

Ah, some methods have documentation:

Note that on multiple calls to poll_read_ready, poll_read or poll_peek, only the Waker from the Context passed to the most recent call is scheduled to receive a wakeup. (However, poll_write_ready retains a second, independent waker.)

But this is a method on TcpStream, not one of the AsyncRead ones

[edgy sequoia]

It is not really possible not to perform the poll_flush call, so I guess my only option is storing the waker from the last caller myself manually and setting it back to that after the call. That sucks.

[nimble cape]

To clarify, there isn’t a global list of wakers, in most cases, each future is responsible for storing any waker that it needs to wake up. Which is sort of clarified in that documentation that you quoted.

When you’re writing futures manually, you need to honour a poll pending and return it back to your caller. Or store the waker and then set your own waker so you can bubble the wake up to the task. That’s what something like join does.

[edgy sequoia]

Yeah, guess I'll store the waker that last did a call to poll_flush which returned Pending and then use that for my call to poll_flush so that it does not overwrite the waker with the one from the read task.

[nimble cape]

You should really wait until the the first future completes (returns poll ready) and only after that should you call poll_flush again.

[edgy sequoia]

Is there a particular reason why I wouldn't want to call poll_flush with the other waker instead? I don't really see a downside

I guess there's an argument for "the other task is already polling it actively, so no need"

[nimble cape]

I’m not sure that it’s storing a waker at all, not if it returns poll ready.

[edgy sequoia]

Ah, yes, that makes a ton of sense. I'm going with that approach then, thanks!

[nimble cape]

Honestly, I think that you’re beyond this level already, but in case it helps, I wrote a series of posts on understanding async/await, the second part is on wakers: https://hegdenu.net/posts/understanding-async-await-2/

[brazen ore]

The tcp stream stores one single waker for the write direction, and one single waker for the read direction. When the tcp stream becomes ready for writing, it calls wake on the waker provided to the most recent call to any of the poll functions in the write direction that returned Pending. Calls returning Ready do not count.