#Backpressure in tower-async

165 messages · Page 1 of 1 (latest)

zealous basin

@chrome mirage hope I'm not using too much of your time (or asking this for the third time or w/e), but it looks like you dropped backpressure entirely in tower-async. Do you have some docs / a writeup on the reasons or smth. similar?

chrome mirage

tower-async was the first iteration on that. Two more iterations followed.

https://github.com/plabayo/rama

is where the latest version of that code lives, directly integrated.

If possible I would love to move back to tower one day, but it will depend how it can evolve.

zealous basin
chrome mirage

Indeed

zealous basin

I'm actually kinda motivated to do the forking now 😄

chrome mirage
zealous basin

Where's the latest version of the service trait specifically?

chrome mirage
zealous basin

Oh, Context looks to be similar to axum's state?

chrome mirage

My reasons for not having ready is because:

  1. all my proxy and web services live behind load balancers, so feels like I don't need it as much if at all in the service code itself
  2. I do appreciate there are use cases where you do want that, but I'm convinced you can have that back pressure directly in your serve code

It's mostly for Dynamic and Static state, but also for some utilities like graceful shutdown and DNS

usage might grow over time, but for now seems fine like it is

The thing there is also that your context starts on the transport layer (e.g. TCP) and moves to HTTP, in rama the services start from the transport layer

zealous basin

Nice I'll listen to that soon

chrome mirage

But honestly this has been one giant distraction that I had rather avoided if I could have. Like at least 1 year delay in my plans 🥲 Such a rabbit hole that I just came out of

But I have huge love for the tokio, tower and hyper ecosystem that where I can I stay close, but sadly for a lot of it I had to drift apart a lot

Originally I'm a C++ developer, who jumped ship to Rust like 10 years ago. The ecosystem is still young. We'll see. My hope is that our ships can re-align again in the future, but time will tell

but it's a giant understatement that rama would not exist without the amazing people that you are all, and the ecosystems with its open source codebases I mentioned before. That would have been way too much for me to do alone otherwise while being a very dedicated father of 3, with a full time job and our own company on the side next to that 😛 So yeah, stretched thin.

As I said, if I could have avoided this 1 year derail I so would have 🥲 And of course technically I could, software is very mallable. But if it doesn't "feel right" then it's had for me to continue going such a path

in the meanwhile happy to talk to you or anyone else as much as you want in the next days, weeks and years about this and more 😄

zealous basin

I am also stretched rather thin but when there's an area where I feel like I can really move things forward, I always somehow find time for it 😄

I have to say the dns thingy in your service trait looks a bit wrong to me (as in, that seems to specific for a general-purpose abstraction like tower). I have wanted a concept of state in tower for more convenient layering in axum though (it's so stupid how there are separate middleware::from_fn and middleware::from_fn_with_state).

Soo I guess I'll try with some sort of middle ground. Once I find a name for my tower fork. Oh and I will definitely do the -> impl Future + Send thing like you did in your latest version 🙂

chrome mirage

Well it's in the context, not in the service trait.
But indirectly it is I guess, and I fully agree.

Rama is still considered experimental despite my actual production use that is building up. But I don't mind breaking for now, which is why I put a big experimental disclaimer.

I need a way to pass it down however, and I haven't found a good way to make it work. Dynamically typed is not working there as I want DNS to be there. But at the same time haven't found a good design yet to make the static state stuff easily extandable.

Related to the latter I do have some stuff like FromRef etc, including macros. But that's for state layers that the user controls, doesn't work as nice when you also want to add external state.

zealous basin

I'll get back to you about this once I have my tower fork set up + a fork of axum on top 😄

chrome mirage 
-> impl Future + Send thing like you did in your latest version

Yeah another idea could be once you can do type Future = impl ... then that might have its own advantages as well.

But haven't found an advantage to it yet, given you can still return Future structs if you need one manually and you should soon also be able to refer to serve::future in stable, you can already do in nightly if I recall

Yeah for sure, being able to use async fn feels amazing, already for that alone it felt great 😛

but there were many other things, e.g. the state passing, the not having to be paranoid about the ready thing that I anyway never used
and some more

zealous basin

Well I'd use type Future = impl Future if it was stable. Kinda nice to support non-Send futures, which this enables. I don't see return-type-notation landing in a while still, and for associated-type-impl-trait there's an open stabilization PR IIRC.

chrome mirage

for tower I get that, but if you are anyway tied to Tokio multithreaded env then I don't see the advantage, as for rama I gave up the idea on being async-runtime agnostic. That's a rabbit hole I also spen some time on

zealous basin

Right

I do like that axum supports non-tokio environments (I think the most commonly used is AWS lambda or sth.)

So I'll keep that

Not sure if that's related to wasm, but that's also to some extent supported, and where non-Send futures become really important

chrome mirage

Yup true true, but I'm kinda waiting for now how the async stuff will mature in next years. As I said I don't mind making breaking releases (respecting semver of course).

It does however mean that you won't build up a big user base as most people get pissed from that. But when I got into Rust it was still pre 1.0 and they broke my code every few weeks, so I guess it trained me to not pay so much value to it 😛 at least if there's a good enough reason for it.

zealous basin

Honestly I don't think people are all that annoyed about breaking change releases once in a while

It's just important to have clear upgrade paths

chrome mirage

individuals maybe not, but most companies do label such software as unstable and stay away from it

zealous basin

Of course there's always that one guy that gets pissed about "being forced to upgrade" because of some third party dependency requiring the new version of your stuff, and doesn't read the upgrading docs 😂

zealous basin
chrome mirage

for rama it's going to be still every few months I imagine, as I need to start using it for plenty of commercial work, so I can stop maintaining dozens of proxy codebases as fully independent code bases with a lot of copy paste, and instead derive a lot of it from Rama 😛 but there is still plenty to improve. The usage both by all my client work and also by external people also helps drives the design, so it goes hand in hand, but it does mean that I will not go for a 1.0 anytime soon

zealous basin

Doesn't sound too bad for me ^^

Of course for a foundational library it would be kinda annoying

But, stuff like the aws libs also have a breaking release every few months AFAIK

chrome mirage

Yeah that's what I am getting at. Tower's success also means it is a lot harder to continue to evolve.

zealous basin

Usually not that much changes so it's not really much of a pain to keep up to date

I don't think its success is why it hasn't been evolving though 🥲

chrome mirage

Well the other factor is having people that are trusted by existed people to move things forward, but that handover doesn't seem to be easy. E.g. I'm definitely not trusted yet. and I do get it, I mean I still have brought little value or work towards these ecosystems so there's no reason to trust me of course with stuff like merging etc 😛

Fully understand that, no issues with it.

zealous basin

I disagree that there's no reason to trust you

In fact I'm very happy to give you write access on GitHub + crates.io once I have that fork is up, if you're interested in yet another distraction 😅

chrome mirage

Just finished removing ServiceBuilder from rama and also synced with tower 0.5 where relevant. Damn, that was not too light on work.

First of all, I do like it very much without ServiceBuilder:

  • ServiceBuilder felt always like duplication both for the developer, as well as having "two" ways of doing thing
  • ServiceBuilder (even with ServiceBuilderExt will always have some layers that have to go through layer, and it's a bit arbitrary, weird to have it mixed like that imho
  • the ServiceBuilder somehow made the concept of layering/stacking more abstract, as a tuple it's a lot more "direct" somehow

So yeah, all in all, love it. But hehe, if you do remove it, you're going to have your work cut out for you, especially in example/test code 🥲

And of course everyone else as well 😛

that's a painful breaking change, but I would say worth it

zealous basin

@chrome mirage reusing this thread to ask your opinion on other tower design things.. How do you feel about the general-purpose nature of tower / the utility of non-protocol-specific middlewares? I believe I've never liked a single middleware from the main tower crate, most of them I never even looked into and others were just a pain to use in the context of HTTP (maybe in part because hyper decided to treat Service-level errors as "abort the connection").

By the way

ServiceBuilder (even with ServiceBuilderExt will always have some layers that have to go through layer, and it's a bit arbitrary, weird to have it mixed like that imho
This is exactly why I don't like it too.

chrome mirage

In rama I took some of these non-protocol-specific layers over and added some specific to rama as well. So I do see the utility in them:

E.g. https://docs.rs/rama/0.2.0-alpha.1/rama/service/layer/index.html

Some of these have generics which make them in the end still protocol specific, but even for those it's nice to be able to share the common logic.

So yeah, don't think I have a desire to delete these.

I think the only ones which I no longer use or need are the Then and AndThen... Going to look tonight into deleting these, didn't know they were still in there. Now that I no longer have ServiceBuilder these are dead ends and duplicates from MapResult and MapResponse.

Either way, I am defintely in favour of common logic where possible 🙂

zealous basin

Which different protocols do you abstract over in rama?

chrome mirage

Transport (is anything that’s stream, eg tcp, tls and soon also others), Http, Connectors (used to establish connections client side), and probably more in future. Might even forgot to mention something here.

Users can abstract over their custom protocols of course as well, same like tower

zealous basin

Okay, but what sort of middleware do you use both in transport and http, or both in transport in connectors, ot http and connectors?

chrome mirage

ConsumeErr. Definitely a useful one for fatal errors specific to that request/connection and that I can do not much useful about. By default it traces, but you can also customise that and choose the kind of replace response.

Hijack layer also used in multiple “protocols “.

Map layers also come to mind.

zealous basin

ConsumeErr and Hijack are inventions of your own?

Haven't heard of anything like that in tower

But at least the first one sounds trivial, i.e. if transport, http and connectors had separate service traits it should be trivial to have copied of those layers for each of the traits, without that being a problem for maintainability

Maybe it would even be an opportunity for better use-case-specific docs

chrome mirage

Yes they are specific to rama

zealous basin

Didn't take me long to go from "I'm gonna fork tower" to "actually maybe I should start from scratch on the lower-level abstractions" 😂

chrome mirage

And I agree that you could also make these specific to protocols.

Seems more like a philosophical choice then really a technical one

zealous basin

Kinda, yes

chrome mirage

Haha, that took me more then 6 months to realise

But given rama is a mono repo that maintainability issue in that regards is either way not there

zealous basin

But you didn't start from scratch, right?

chrome mirage

No, but given it took several times starting rama from scratch , including many revisions of the design, it amounts to the same work as scratch.

So dunno how you would label it. In the end it can be seen as a fork. As that’s how it started. Be it a permanent fork

I liked from the start the elegance of tower. I compare it often to a next step above introducing routines/functions.

So there was not much else imho to change about that, other then to attach a context to it and ripping out the ready stuff. And anything else we discussed about

That said it’s all open ended so looking forward to see any code produced by you and others. Everything can be better. And I’ll gladly pivot to it

zealous basin

I once used to admire "elegant" abstractions

Nowadays I'm very very happy to have switched from Haskell to Rust.

In the former it feels like everybody invents their own abstractions, and while that can lead to super terse code and inventing your own abstractions can give you a lot of insight into a given problem domain, ultimately I find average Haskell code borderline unreadable.

I blame that a lot on "invent your own operators" as well, but some of it definitely is the preference of extremely high-level one- or two-line combinators over copy-pasting a bit of code

chrome mirage

Agree. There has to be a balance. But where that balance is,
is of course subjective

For me I found that in Rama, but for many that will be elsewhere

Haskell was either way meant for academic research, boggles my mind that people took it into production levels. Then again the latter helps drive the first to a given extend as well

zealous basin

Haha idk the details of Haskell's history but it is does indeed feel crazy that people took it to production instead of building more pragmatic languages that take some inspiration from it, but are more pragmatic in others

I wonder what the next big language will be that takes some of the elegance of Haskell / ML (?) but mixes in a lot of pragmatism to escape the academic world

In some ways Rust matches that description, but the low-level aspects are completely orthogonal to me

chrome mirage

Protocol agnostic wise the most interesting in Rama is the service/layer/state stuff. The common layers are an extra but not the cool bits.

Most interesting stuff happens in the protocol specific modules as well as how these networks layers come together in various ways.

To come back to your original question

So not important, but still cool enough that I do have such agnostic layers.

zealous basin

Hmm, I don't think I get what you mean

You mean it's cool that you only have to define two (or however many) traits for all of (connectors, http services, transport services)?

Plus their respective blanket impls (like the tuple thing)?

(looking at tower-layer and tower-service there are really only a few impls, tuples + identity + reference for layer + box for service)

chrome mirage

That is one aspect yes. It makes it fairly simple to know what’s going on. As the code flow is in your face. In contrast to many frameworks where the code flow is pretty abstract

It makes it also modular and reusable. Out of the box.

And you are never blocked on something rama does as you can replace anything with your own code. Where you do so is up to you

zealous basin

Hmmm I get what you mean but I think separate HttpMiddleware, HttpService, TransportMiddleware, TransportService etc. traits would increase clarity

This is the kind of thing I'm currently considering experimenting with next

chrome mirage

Maybe. But that’s more again philosophical and how much you like generics

Personally i don’t think it would improve clarity.

There is an issue with clarity at compile time, but it’s not that

zealous basin

But I really dislike abstractions that don't serve a clear purpose

And I'm 95% convinced now that an abstraction like tower that's as general as "async request to response" is just too abstract

I.e. an abstraction that doesn't "pull its own weight".
I think copy-pasting the relevant reusable code once per lower-level (protocol-specific async request to response or similar) abstraction is going to be better.

But I think I already made that clear before. I don't mind that it's "more philosophical" (than what, actually?) ^^

chrome mirage

Yes yes. I get you. It’s certainly not what rama makes rama. So happy so to see where your exploration leads and see how it might influence a future version of Rama

Either way I’m your cheerleader 😉

chrome mirage

Thinking about your idea though. I’m not sure what a “layer” specific Service trait would do.

Layer is even worse of an argument, but let’s focus on Service.

A transportService would still require a Stream generic trait instead of a Request. As each layer can and often does change that type.

On connector that’s especially true, be it a child generic. And similar for http where the Body is still generic.

So you wouldn’t remove the generic. But you would have explicit name and documentation. So there is a point in your favour. But dunno, still not convinced.

Plus adding traits comes with its own cognitive load, as users will now need to learn 3 or more instead of one, all while still having to work with the same kind of trait bounds and despite no interface difference between these traits.

zealous basin

I've got ServeDir and ServeFile from tower-http ported 🙂

If you want early access to the repo (private for now) let me know

chrome mirage

Sure, happy to follow the progress

chrome mirage
  • So you basically have a Serve trait focussed on Http. I do like how you have so few trait bounds on there. Does that work nicely within a tokio work stealing multithreaded env?
  • Either is a life saver I noticed in Rama. Helps a lot in reducing generic stack depths and no boxing. I allow it up to 9 variants I think and with a macro to allow easily to implement either support for your own trait. Really useful once you know about it
  • Layer renamed to middleware. Makes sense even though by now I get used to layer 😄

Beyond that seems like the fun is still to begin for zon.
Why the name btw? 😛 It means "sun" in my mother tongue 🥲

zealous basin

Oh that's your mother tongue? 😄

chrome mirage

Dutch? Yes. Well the Flemish variant, but close enough.

Right now learning Norwegian as my 5th language. For someone knowing Dutch and English it feels like a fairly simple language to learn. Of course I say that very ignorant still being on a pretty easy stage, haha

zealous basin

I was looking for an easy to remember, short name not taken on crates.io and after a few other idea that didn't pan out, I looked at words around a solarpunk theme (because solarpunk is awesome), including in other languages because foreign words often sound cool if you pretend they're English 😄

So that's how I arrived at zon 🙂

chrome mirage

haha I wonder how English native people would pronounce that 😄

zealous basin

I'm thinking like a mix of "zone" and "son"

chrome mirage

nice, has a nice ring to it

zealous basin

How is it pronounced in Dutch?

chrome mirage

Actually pretty much like that, maybe that's why I like your proposition, haha. I'm such a fool

zealous basin

My most recent though (yesterday evening) about the API design was that I really don't care to have a "service" and "handler" split like tower / axum

chrome mirage

It's the z sound of zone. Then the "o" sound like when you say bob and then the n of son Combine those 3 "sounds" and you get zon.

zealous basin
chrome mirage

Coolios

The service/handler part I don't get though

zealous basin

I'm gonna make functions like those that implement axum::handler::Handler implement zon_core::HttpService

So async fn(http::Request<B>) -> http::Response<B2> implements zon_core::HttpService<B, _, ResponseBody = B2>

(the _ there is going to be a "dummy" trait parameter like Handler has, defaulted to () to make manual impls of HttpService not look weird)

chrome mirage

Ah yes I also do that in some places. It does get weird if you want to support combinations, but only so in the documentation, actually writing the code is still natural

Yeah lol rama is full of those and more

I sometimes feel more like playing with lego then a programming language

zealous basin

Heh

chrome mirage

Most of the time it is ok

  • but every 6 months I do run into cargo check being stuck on something
  • and it can give really insane errors

TLDR: gl to any potential users 😵

Did learn a whole lot and as Rust does keep improving my hope is that al ot of this can be simpler as the language and rama both evolve

🤞

zealous basin

Honestly I doubt there will be major advances w.r.t. that ugly coherence bypass 😄

But fortunately proc-macro workarounds like axum's debug_handler are possible

chrome mirage

There is an effort ongoing to give library authors more power to provide custom errors and diagnostics AFAIK

zealous basin

Right

We use that in axum

But the only useful thing it does is point people towards debug_handler 😄

chrome mirage

but that effort is still only starting, pretty certain more stuff will come as we go towards the next edition and beyond

yeah for now I haven't ported that macro yet

might need to figure that one out in worst case at some point

zealous basin

Hmm yeah I guess we can hope for more diagnostic attributes

chrome mirage

Was under the impression that that was seriously worked on

but 🤷‍♂️

will also be happy when the parallel frontend stuff comes to stable rust

have tried it out and it makes a giant difference

in terms of tooling speed

And I'm saying that already being on a monster macbook pro M3 dev machine, which already makes a big difference as well

zealous basin

I know diagnostics stuff is being worked on, not sure about work specifically for coherence-defeating traits

chrome mirage

Maybe not, but luckily that's not an issue you run much into once you are used to it. But probably every new person will run into it ... t.t

zealous basin

I guess it's still one of the most frequent problems around axum, but (a) it seems to be becoming a little less common and (b) it's not like it's magnitudes more common than other problems

So I think it'll be fine 😅

chrome mirage

Seems I hit a rustc looong compile time error. Going to try to box the connector in the culprit examples/integrations tonight as a tmp bandaid…

But if you somehow have an idea @zealous basin would be cool: https://github.com/rust-lang/rust/issues/129844

Saw in the tracker of rustc that you and David hit something similar with axum at some point. But if i understood correctly, it was never really fixed? Just boxed?

zealous basin

Also you can "simplify" stuff like

    L: Layer<S> + Send + Sync + 'static,
    L::Service: Service<State, Request<Body>, Response = Response>,
    <L::Service as Service<State, Request<Body>>>::Error: Into<BoxError>,

to

    L: Layer<
        S,
        Service: Service<
            State,
            Request<Body>,
            Response = Response,
            Error: Into<BoxError>,
        >,
    > + Send + Sync + 'static,

nowadays (unless you have a couple releases old MSRV) which might improve compile times

chrome mirage

My MSRV is 1.80 😛

Just because why not. I can count users at my hand and most I support closed source.

Will read. Thx!

In the meanwhile starting to split crates, as it might make community crates lighter as they can just pull in what they need.

Comes with its own challenges, especially the orphan rule.

At the same time it seems to make some of my design better, got through the biggest refactor already, and looks better already.

That compile time is still forever. So yeah going to read up!

The crux of the problem is "gee grandma, what big types you have there", because...

Lol that hit home. Do not look at my service types. Don’t shame me 😦

zealous basin

Heh

By the way, should we move this to DM? 😅

chrome mirage

Maybe. Or a channel for tower spiritual conversations.

Either way fine for me