How to return struct containing `X` along with a box containing another struct with a borrow of `X`

[broken harbor]

I am trying to use a crate that has a very.... interesting ownership model.

Really, all I actually want to make work is just:

struct RTTClient<'a> {
  decoder: Box<dyn StreamDecoder + Send + Sync 'a>
}

impl<'a> RTTClient<'a> {
    pub fn new() -> Self {
        let table = dumb_crate::create_the_table();
        let decoder = table.new_stream_decoder();
        
        Self {
            decoder
        }
    }
}

The problem is, table.new_stream_decoder returns a struct with a permanent reference to table:

See here

pub fn new_stream_decoder(&self) -> Box<dyn StreamDecoder + Send + Sync + '_> {
    match self.encoding {
        Encoding::Raw => Box::new(stream::Raw::new(self)),
        Encoding::Rzcobs => Box::new(stream::Rzcobs::new(self)),
    }
}

And, for reference

impl<'a> Raw<'a> {
    pub fn new(table: &'a Table) -> Self {
        Self {
            table,
            data: Vec::new(),
        }
    }
}

The result is that table gets dropped when RTTClient::new returns.

So, my immediate thought was: Let's have RTTClient take ownership of table, that way it's not dropped:

struct RTTClient<'a> {
    decoder: Box<dyn StreamDecoder + Send + Sync 'a>,
    table: Table
}

impl<'a> RTTClient<'a> {
    pub fn new() -> Self {
        let table = dumb_crate::create_the_table();
        let decoder = table.new_stream_decoder();
        
        Self {
            table,
            decoder
        }
    }
}

That doesn't work, either, though. Because decoder has borrowed table, I am not allowed to move it into RTTClient.

So, great, the way decoder is constructed has prevented me simultaneously from allowing table to drop and from preventing table from dropping.

I'm stumped -- What on earth is one supposed to do in this situation? I feel like there MUST be a good solution to this, but I've been pondering this a couple hours now and still am coming up short.

The only usage example I have here is another crate (probe.rs) and IIRC they just straight up do an unsafe transmute to bypass this issue, which I don't wanna do.

[onyx ivy]

this is actually hard, kinda

you cannot do it with a simple Rust struct containing table, decoder. this is for at least two reasons:

• the reference to Table would be invalidated any time you move the RTTClient, such as by returning it from new()
• there is no language rule that can stop you from invalidating the reference by accessing the table field while the decoder exists

you have to use special "self-reference" tools to handle this

any of https://docs.rs/yoke/, https://docs.rs/ouroboros/, or https://docs.rs/self_cell/ will do

another possibility applicable to some situations is to put your two things in an async block (async handles selfreference through Pinning), but that is mostly only useful in the same cases that you could also solve your problem by spawning a thread to own and manage the two things (see "actor pattern"), except that async would let you avoid having the resource costs of the actual thread

[broken harbor]

blehhhh what an unfortunate design choice by this crate

[onyx ivy]

since this is a “stream decoder” you should think about whether the thread/async approach is a reasonable option

basically you write a function that loops putting data into the decoder from a channel, and taking frames out and writing them to a channel

then that function can be run on a thread or an async task

[broken harbor]

I may indeed wind up switching to async at some point for this project, but I'd prefer if the API for this RTTClient were synchronous

[onyx ivy]

you can run an async task internally in a way which is invisible from outside, for this purpose, but you might not want to because at that point yoke and friends are less boilerplate

[broken harbor]

yeaaaah I'd rather avoid pulling in async just to get around this

[onyx ivy]

the upside of the async approach would be you are able to say "I am using no unsafe shenanigans"

the selfref libraries all have some kind of unsafe inside. in the ones I named, it is well-tested sound code (nowadays; the history of selfref is fraught), but some people would like to avoid that entirely

[broken harbor]

honestly just on principle the idea that this crate has cornered me into either invoking a strange async wrapper or some kind of unsafe wrapper genuinely just irks me

[onyx ivy]

reasonable! I have been known to say "that's a design flaw in the library"

it should offer you the option to use something other than an & borrow

[broken harbor]

agreed, that seems just unhinged to me

I wonder if I could get around this by moving the RTTClient struct onto heap somehow?

[onyx ivy]

it's also arguably a design flaw in Rust that it doesn't offer you safe selfref tools

[onyx ivy]

sure, ouroboros does that … and you still need to solve the rest of the problem

basically all selfref requires either something Box-shaped or something Pin -shaped

ouroboros is taking the approach of boxing stuff so it doesn't move, but you still need the safe API around that

it can't be simple at the language level because the borrow checker doesn't understand the lifetimes involved (except when they are expressed in an async block)

[wind siren]

the most common solution to this is to just do two-step init, where instead of initialising everything in one function, you have the caller first create the source value (your table) and then pass a borrow to that to another function that returns the borrowing value (your decoder)

i would not recommend looking into self-referencing types unless you absolutely have to

[broken harbor]

I thought about doing something like that, though I am worried the lifetimes might still get in my way if I try to store the result of the second stage in RTTClient -- I think that might require a heap allocation at that point.

Though I guess I am already kinda at the point of needing such a thing

am I wrong about that?

And if I'm right, and a heap allocation is necessary, then I start to wonder if a heap alloc would let me do it in one step after all

[wind siren]

i can't really say, not sure where you're thinking that a heap allocation might be needed, but you're already doing an allocation to create your boxed stream

[onyx ivy]

you will always need either:

• at least one heap allocation, or
• storing the parts in separate let variables or similar
  in Pin terms, these two strategies correspond to Box::pin(value) and pin!(value)

[broken harbor]

That boxed stream heap allocation is not my choice -- the crate does that

[onyx ivy]

(afk)

[broken harbor]

to avoid XY problem, what I really want here is any non-unsafe, non-async way to store the result of table.new_stream_decoder (or a reference to it) inside the RTTClient struct

But yeah I guess I see your point, that is heap allocated technically already

[wind siren]

unless you're talking about self-referencing types i wouldn't generally expect heap allocating the source value to help

[broken harbor]

I think I am technically dealing with a self-referencing type here, since RTTClient needs to own table in order for it to live, and decoder, by the very nature of its construction, contains table: &'a Table

[wind siren]

well not if you do the two-step init variant, where you have the caller create (a value wrapping) the table before constructing the RTTClient, so the table can just live in its own variable outside the client

[broken harbor]

yeeeaaah see I was hoping to let the decoder live inside the client

or at least let client be the only entity with a reference to the decoder

[wind siren]

in essence you'd have one function wrapping create_the_table and another wrapping new_stream_decoder, so the caller api mirrors how you initialise it under the hood

[broken harbor]

I'm getting the picture, yep, but frankly my entire motivation for creating this struct was so that I didn't have to be juggling both entities (and a bunch of other internal details) when contacting RTT endpoints

[wind siren]

unless you have an unavoidable reason for why you absolutely cannot do it in two steps, i would recommend staying very far away from self referencing types

it's one of the biggest cans of worms rust has to offer

[broken harbor]

I don't want to do self-referncing types, I also don't mind doing two steps, however having RTTClient not maintain at least a reference of some kind to the decoder does kinda defeat one of its major purposes

cause think about my usage pattern here: Every subsequent call to RTTClient is going to require me to pass the decoder to it

[wind siren]

no i just mean that the caller should pass a reference to the table to RTTClient::new, the RTTClient can still store the decoder internally

[broken harbor]

that just screams "why can't RTTClient just own the decoder?"

the RTTClient can still store the decoder internally

Ah, okay, then we are on the same page. Except I have been trying to do exactly that during this whole convo, and still having issues

Let me translate what I've got now, one sec

[wind siren]

you mean that this doesn't work for you?

struct RTTClient<'a> {
    decoder: Box<dyn StreamDecoder + Send + Sync + 'a>
}

impl<'a> RTTClient<'a> {
    pub fn new(table: &'a Table) -> Self {
        let decoder = table.new_stream_decoder();
        
        Self {
            decoder
        }
    }
}

[broken harbor]

That's not what I tried, no

But notably that approach comes at the expense of RTTClient also not being able to construct the table,

[wind siren]

yeah

hence the two-step init

if you need control over how that table is initialised, you should have another type wrapping the table too

[broken harbor]

I just want the "user" (me in other places) to be able to initialize an RTTClient and not have to deal with intermediate values

[wind siren]

struct MyTable {
    table: Table,
}

impl MyTable {
    pub fn new() -> Self {
        // do init here
    }
}

struct RTTClient<'a> {
    decoder: Box<dyn StreamDecoder + Send + Sync + 'a>
}

impl<'a> RTTClient<'a> {
    pub fn new(table: &'a MyTable) -> Self {
        let decoder = table.table.new_stream_decoder();
        
        Self {
            decoder
        }
    }
}

you can even have a helper method on the Table wrapper like

impl MyTable {
    pub fn new_decoder(&self) -> RTTClient<'_> {
        RTTClient::new(self)
    }
}

so you can just do

let table = MyTable::new();
let decoder = table.new_decoder();

wherever you need to use it

[broken harbor]

Yeah that last thing is kinda exactly what I'm trying to avoid xD

[wind siren]

the alternatives are worse in 99% of cases

[broken harbor]

It feels instinctually very wrong that this is impossible

I may simply not bother with RTTClient in this case, if I am truly forced to do it as two totally independent structs

[wind siren]

it's the cost of requiring memory safety while also not having a garbage collector

[broken harbor]

It feels instinctually wrong that this should require a garbage collector to be safe

[wind siren]

in languages with garbage collectors, the gc can just update any self-referential pointers if the memory moves

but rust doesn't have managed memory

[broken harbor]

I am well aware

But IMO managed memory should not be needed for something like this

[wind siren]

well idk what to tell you, the standard solutions are either to have the gc handle it, or to do what c++ does and have move constructors that manually update self-referential pointers when the value is moved

[broken harbor]

Maybe that is my naivete, but the fact that I could then say

let wrapper = Wrapper {table, decoder} and pass that around to a bunch of functions that use either table or decoder, it just feels like this is a special case where it ought to be possible to specify the lifetimes correctly

[wind siren]

but the c++ approach is inherently unsafe

so however rust ends up solving it natively in the end will necessarily be a novel approach

and figuring out a design that is both ergonomic and sound isn't easy

[broken harbor]

I feel like just being able to specify that table should live in the calling scope (like what implicitly happens for a typical return value) ought to completely solve the lifetime issues that prevent this

[wind siren]

super let is an idea that people have been discussing to do that fwiw

[broken harbor]

ahhhhhh

[wind siren]

https://blog.m-ou.se/super-let/

but having it work across function calls and not just scopes within a single function is less straightforward

[onyx ivy]

I disagree with oklyth. Rust could handle this problem without any garbage collection; it just has not been designed or extended to do so, yet.

[wind siren]

i don't think it couldn't, i just think it's a hard problem

i know that native pinning has also been discussed in the context of in-place initialisation

https://hackmd.io/@aliceryhl/BJutRcPblx

[broken harbor]

ugh yeah it is such a nightmare to move the table initialization out of RTTClient

I significantly simplified the code here in my examples

In the real code, table relies on parsing an ELF file, and there are other special variables the RTTClient extracts for that too

I guess I will just make RTTConfig containing the table and those other special values

Also, even worse, I wanted a wrapping type around RTTClient which would contain two of them, and this means the user is going to have to provide RTTConfig for both of those when using that dual client

In fact this makes that dual client wrapper completely impossible to implement cleanly

And I do really ahve a legitimate reason for wanting to do that: I have to spawn two separate clients to be able to decode RTT data from the bootloader AND post-bootloader firmware for this board

[onyx ivy]

I think you should try using yoke or ouroboros

the whole point of these libs is to encapsulate the shenanigans and make a thing that solves your problem

[broken harbor]

yeeeaaah maybe you are right. I was worried about readability / having to justify that to my reviewers, but the headache doing it any other way is about to cause I think makes it worth it

[onyx ivy]

the hazardousness of self-reference abstractions is mainly around generic self-reference. your problem is not generic

[wind siren]

doesn't yoke have known soundness issues

[broken harbor]

I am prolly gonna use ouroboros anyways, it's more readable

[onyx ivy]

hm, was not aware of that

ouroboros is careful about noalias though

[broken harbor]

I also just like that ouroboros explicitly declares cross-references, rather than relying on a "covariance" concept

[onyx ivy]

yoke separates the borrowing thing (generic parameter Y) from the referenced thing ("cart", generic parameter C), instead

if anything, it's more explicit than ouroboros by forcing you to put the things in one of 2 buckets

[broken harbor]

Fair enough, I guess I just find the ouroboros way more attractive because it more closely matches the topology I'm somewhat forced into by this unhinged crate