Idea for better serialization framework

[wraith sequoia]

Interesting. So if that's the case, where do the protocol container types play into the grand scheme? (I.e. ValueProto)

[thorn badger]

Those are higher ranked types that don't have the lifetime

[thorn badger]

All the protocols are of the form dyn Trait<'ctx> + 'a (this is what the visitor and walker values coerce to) however tracking these lifetimes in every usage is annoying, so I use higher ranked types to erase the lifetimes when they aren't needed

[wraith sequoia]

What does that look like?

The erasure, specifically

[thorn badger]

every dyn Trait<'ctx> + 'a is given a unique TraitProto without any lifetimes. this can then be used for things liek getting a TypeId

[red kettle]

does yeeting async generics make anything easier, or is all the complexity already there once you need lifetimes

[thorn badger]

they are orthogonal

[red kettle]

cool

[wraith sequoia]

a big question i have about AnyTrait, why is Available implemented as a tuple of the available types? How do you utilize that?

[thorn badger]

I put it there for documentation purposes

is how it looks

I don't know if I will keep it

I will probably have it add docs instead

it doesn't do anything functional in the code

[modern jetty]

wait how are you doing GCE

or is that just rustdoc

[austere saddle]

what the hell

[dense oasis]

This would probably be a lot easier to read if rustdoc formatted it on multiple lines

[south terrace]

"easier to read if formatted better" you speak like anyone understands a single world in that type alias tbh, whats sequence proto? hint proto? tagdyn? tagconst?

edited because it sounded rude

[modern jetty]

You can click on them, as they are links, which explains more.

[dense oasis]

You did just try to understand a complex library based on nothing but a single picture

What did you expect would happen

[south terrace]

Yeah, but the codebase has very little documentation, so its kind of still hard.

[modern jetty]

the implementation isn't even done, why are you expecting in-depth documentation lol

[south terrace]

touche, i was simply just commenting on it being more readable if rustdoc formatted it better. i dont think anyone is confused by the formatting.

[south terrace]

i am really not really expecting anything tbh

[modern jetty]

A (docs) is bad
B (formatting) is bad

A and B are not mutually exclusive, aka, B can be bad and A can be bad.

[south terrace]

okay, cant argue with that

[thorn badger]

For the record this is one of the reasons I haven't published yet 😅, I don't feel like having a monstrosity of a crate without at least okay docs to go with it.

[thorn badger]

Also, as explanation, everything with Proto in the name are higher ranked types for the protocols, a protocol being a trait used in a specific way. The Tag types are either compile time or runtime shared types based on string identifiers (the .to_int() encodes it in a way for rustc to understand)

[thorn badger]

I would prefer it to have links to the items, but having it formatted like this is ... not fun

[thorn badger]

They aren't generic const expressions, just const expressions

[modern jetty]

oh huh, yeah

[thorn badger]

Also I don't expect an average user to ever touch this part of the crate

[modern jetty]

wait I might be able to use this

holy shit I can

[mellow thistle]

woo

[modern jetty]

thanks for teaching me a trick, I can probably make aformat better now

did Not know you could turn a type into a const generic that easily lol

[thorn badger]

Fair warning that it makes rustc errors incomprehensible

Because it just starts dumping integers at you

[modern jetty]

I am already doing typenum stuff, errors became incomprehensible ages ago

[modern jetty]

Ah fuck, improving aformat with this requires const traits, as converting typenum types to integers is implemented via the Unsigned trait, not a direct implementation on the UInt types.

[thorn badger]

[wraith sequoia]

So basically, Walker drives Visitor, and Visitor contains a protocol trait. But how does the protocol know what to do? What does the visitor react on to drive the protocol? Unless the two are more unified than I thought, and the Walker is fully aware of the protocol that it's driving

[thorn badger]

So the process is the walker does a lookup for trait A (the protocol) on the visitor. And the visitor can either return a trait object for A or it can not

[dawn herald]

I cannot wait to be able to directly convert one struct to another without either of them having to know TryFrom exists :)

[thorn badger]

So if the walker wants to use the Value protocol to transfer a rust value it looks that up on the visitor and if the visitor returns a trait object for it then it calls the visit method on the trait object

[thorn badger]

I do have some basic examples of this in treaty right now

[mellow thistle]

ohh show me

[thorn badger]

Also Visitor itself is usually the implementer of the protocol trait

[thorn badger]

https://gitlab.com/konnorandrews/treaty/-/blob/main/src/lib.rs?ref_type=heads#L272

Much easier than writing a format for testing

[dawn herald]

So Visitor knows what protocols it implements, and can pass one of those protocols to Walker when asked

[thorn badger]

Yep

[dawn herald]

let y = Y::build_async(x.as_async_walker().await).await.unwrap();

lol so many awaits

[thorn badger]

Yeah in the async environment everything becomes a future

[dawn herald]

[wraith sequoia]

Oh my god this entire library makes sense to me now

[thorn badger]

See it's simple

AnyTrait (which does the trait object lookup) is really the heart of this technique

Everything else is just extra fluff I added for other features

[wraith sequoia]

https://tenor.com/view/always-sunny-frank-oh-my-god-i-get-it-gif-22328165

[thorn badger]

This is how treaty gets around needing a data model

[austere saddle]

what a clever thing

[mellow thistle]

?crate never_say_never

[tribal rampart]

never-say-never

The never type (the true one!) in stable Rust.

Version

6.6.666

Downloads

727 536

Jan 19, 2022, 2:10 PM

[thorn badger]

Yeah that's a yandros classic

[modern jetty]

gotta love 700k downloads

[thorn badger]

It's mainly from polonius-the-crab

[mellow thistle]

which in turn is mainly from specs

[austere saddle]

ngl this guy is crazy

[thorn badger]

Yandros helped me out quite a bit with treaty's cursed internals

[modern jetty]

and me with aformat

[thorn badger]

Like the bijective higher ranked types

[mellow thistle]

please tell me there's at least one amogus in a macro somewhere

[thorn badger]

Not yet

[austere saddle]

He also helped me when I was doing cursed shit with macro. (I failed tho)

[thorn badger]

When did we get a cat plead emoji!

[austere saddle]

one more fancy thing from type theory to study (sadge)

[mellow thistle]

it's from ROUS

[thorn badger]

[quaint raft]

if we ever get const traits i think i'm going to organise a party to commemorate that

[thorn badger]

I also love that it's a turbo fish

[modern jetty]

the improvement would have allowed me to remove the const INDEX thing in typenum_mappings

[austere saddle]

Why don't we have a ! in the std and need a strange method to get it?

[modern jetty]

but nope, still gotta generate this lol

[thorn badger]

It's not stable yet

[modern jetty]

it's not meant to be usable in stable in any place other than function return

[thorn badger]

The above only works due to a bug in rustc

Which leaks it into stable

[modern jetty]

never_say_never is exploiting a bug which now cannot be patched because of the whole... 700k downloads thing

[austere saddle]

what does "unstable" actually mean?

[modern jetty]

not usable unless on a nightly compiler and using #![feature]

[thorn badger]

It could change at any time

[mellow thistle]

can be changed without a new major version

[austere saddle]

that makes sense

[mellow thistle]

so for std etc, it means it can never change

[austere saddle]

but it's unstable so it can be changed at any time, isn't it?

[austere saddle]

this message makes no sense to me

[thorn badger]

On a side note, I am impressed we have gotten this thread past 2000 messages

[mellow thistle]

I meant if something is stable, mb

[rough oasis]

on another side note: "due to"

[thorn badger]

?ban @rough oasis

[tribal rampart]

Banned user bruh791

[rough oasis]

[austere saddle]

ah

[modern jetty]

theoretically, never_say_never could be broken and that's entirely in the rights of the stability policy... but it would be quite disruptive so now there is just this leak in the stablity system lol

[austere saddle]

how didn't they notice it

[modern jetty]

because it's pretty easy to forget that you can extract the return type out of a function into a type alias

without the bug, trying to use ! would lead to rs error[E0658]: the `!` type is experimental --> src/main.rs:31:14 | 31 | type Never = !; | ^ | = note: see issue #35121 <https://github.com/rust-lang/rust/issues/35121> for more information
on stable, or the following on nightly rs error[E0658]: the `!` type is experimental --> src/main.rs:31:14 | 31 | type Never = !; | ^ | = note: see issue #35121 <https://github.com/rust-lang/rust/issues/35121> for more information = help: add `#![feature(never_type)]` to the crate attributes to enable = note: this compiler was built on 2024-06-21; consider upgrading it if it is out of date

[quaint raft]

i think they meant how it was allowed to go unnoticed for so long

[dense oasis]

It's not broken enough to break, and this code should work as soon as we stabilize the type

[thorn badger]

To be fair it's almost impossible to have it not happen

[austere saddle]

every day comes to me with new knowledge about Rust...

[modern jetty]

of course lol

[thorn badger]

You should read my blog

[austere saddle]

link?

[thorn badger]

https://konnorandrews.gitlab.io/descent-into-madness/

I need to post something new

[austere saddle]

thanks

[zealous frigate]

Tag "🐸" successfully created.

[thorn badger]

-🐸

[zealous frigate]

https://konnorandrews.gitlab.io/descent-into-madness/

[mellow thistle]

💢🐸

[austere saddle]

post about higher ranked types and trait bounds

[thorn badger]

I'm not sure if I'm ready for that one yet

[austere saddle]

you must post when you will be ready

[thorn badger]

I might split it into parts of increasing difficulty

The full versions that treaty uses are a few steps beyond the ones even yandros has

As they mix GATs into them

With things like late where bounds and implied bounds

And then the bijective ones need a different setup

So yeah probably going to need multiple parts

[dawn herald]

The fact that you can lie to rustc in a function's body by using for<'a> in a where bound is funny

[thorn badger]

[dawn herald]

Alas, rustc gets the last laugh: trying to use the function checks the bound and causes a compile failure

[thorn badger]

I left that out as an exercise for the reader

[dawn herald]

Hmm

I should make a blog

[thorn badger]

I got bullied into this /s

[dawn herald]

Oh?

[thorn badger]

Everyone wants me to explain how stuff in treaty (and my other crates) works

[thorn badger]

You can join the writing club then

[dawn herald]

Hmm

[long sigil]

I do not want you to explain treaty

[thorn badger]

So black box?

[dense oasis]

But I do

[long sigil]

@thorn badger what's your hourly rate

[thorn badger]

Minimum is 6 bugs

[long sigil]

I can introduce 6 bugs an hour

[wraith sequoia]

can we pin this

[wraith sequoia]

hehe GYATS hehehe

[austere saddle]

does yandros has a blogpost or something about all this stuff?

[thorn badger]

That's a good question

[dawn herald]

Yandros is cursed and that's great

[thorn badger]

I know yandros is in the writing group, let me check

[austere saddle]

like I have no idea from where yall getting all this interesting stuff (htk hrt and etc)

[thorn badger]

We can blame yandros

[austere saddle]

I just found this https://www.youtube.com/watch?v=BdXWlQsd7RI idk if it will help me

[thorn badger]

https://docs.rs/higher-kinded-types/0.1.1/higher_kinded_types/trait.ForLifetime.html is my recommendation for higher kinded/ranked types and how they apply to rust

This is yandros' main writeup of it to my knowledge

[austere saddle]

I will read it

[wraith sequoia]

So just to check if I've been reading the code right, more or less, AnyTrait will map a type based on its TypeId with upcast_to_id, regardless of TypeNameId for brevity. The complexity comes from the idea that a protocol (for this example I will use a protocol trait Poo) has a "container" PooProto that implements the TypeName::MemberType trait, indicating that type T = dyn Poo. So, you use the TypeId to map the type, and the T to initialize the AnyTraitObject, right?

I'm not 100% what I've said makes sense, I'm still trying to parse it out

[thorn badger]

Basically

TypeId only works on static types

So we need to erased the lifetimes to get one

So if we have say dyn Trait + 'a and we want a TypeId, we would want to take it on dyn Trait + 'static as that's a static type

Hence we have removed the generic lifetime by replacing it with the static one

How treaty does it is more complicated to have the type system prove this is okay to do

(for those curious the lifetime isn't completely erased there is some unsafe code that makes sure it stays consistent, which does mean this operation is invariant)

[wraith sequoia]

Is that the Indirects/RawIndirects?

[thorn badger]

Those are involved yes, the TypeNameId's lifetime is a witness of the original lifetime the type had

Which AnyTrait then uses to have the borrow checker prove it's okay by giving it what it had before

We sort of cut out the lifetime and then merge them back in

(2 lifetimes in treaty's case)

Or well sorry AnyTraitObject is what does the cutting out then reinjection

In previous versions I had TypeNameId carry the lifetime

[wraith sequoia]

jesus christ

[thorn badger]

It also doesn't use static because that doesn't always work, treaty's traits can have lifetime containing generics which doesn't work if we blindly swap the explicit lifetime for 'static

Instead we generate a unique uninhabitable type that is 'static that we associate with the lifetime containing type in such a way as the type system proves its impossible for two types to use the same struct marker

Figuring out if that was even possible took me a long time

And finding a bug in nightly rustc

Oh also for the record, none of this lifetime higher ranked stuff can be done with GATs

It needs to be done using for<'a> do to the issue lifetime GATs have

[rough oasis]

due to*

[dawn herald]

Was this the OOM?

[austere saddle]

what is GATs?

[thorn mantle]

Generic associated types

[visual tartan]

Why do you have hkt

And Higher ranked types, insane stuffs

[thorn badger]

I have generic types that are generic

[austere saddle]

honestly hkt actually makes sense to me

but hrt aren't yet

[wraith sequoia]

For Invariant, why did you pick PhantomData<fn(&'a ()) -> &'a ()> instead of say PhantomData<&'a mut &'a ()>

?*

[dense oasis]

There's not really any reason to prefer one over the other

[wraith sequoia]

Okay. I wonder that frequently when looking at code that uses PhantomData

[thorn badger]

I feel like the fn ones are more apparent in what they are doing

[modern bone]

Then again, it's also being named Invariant which makes it entirely clear what it's doing

[thorn badger]

I mean more of how it's making the lifetime invariant

[wraith sequoia]

I was going to ask about Marker too but I thought I'd save the question about its mechanics later, since your paragraph yesterday indicated there's more at play than just making an invariant lifetime

[thorn badger]

Marker is my PhantomData that doesn't effect the type holding the marker

Like if you have a !Send in a Marker then the holding type doesn't become !Send

[wraith sequoia]

Oh, that's cool

I thought it had more to do with the whole matching identities thing I vaguely remember

[austere saddle]

that's really cool

[thorn badger]

Marker can also hold unsized types without effecting the containing type

[wraith sequoia]

@thorn badger You know, I don't know if this is a question with an obvious answer, but I've been thinking about how you would go about formulating a treaty protocol that does actual off-the-wire deserialization. Wouldn't you need to formalize a different protocol/visitor for each type, since they all have different ways of deserializing a type in their format?

[thorn badger]

So treaty provides the core protocols for general types, we then define flows for how a type should use those protocols to talk to each other. So if a walker and a visitor follows this they will have maximum compatibility. The way this works for most things looks like serde's data model

So for example if a format has a integer it would try the Value<i32> protocol to give it to the visitor

If a format wants to it can define better protocols that it would prefer to use over the default ones

[wraith sequoia]

I guess my question is like, say you're using a format that deserializes i32 and i64 differently. Would you implement your own Value type that matches the TypeId of types being visited, or would you make different visitors like FixedLeI32 or FixedLeI64

[thorn badger]

So the walker should use whatever it has, the integer builders actually support all the integer types and attempt to cast the value if it fits

If the walker wants to know the endianness the specific integer wants them yeah that would need to be done either via a new protocol or by a hint tag

[wraith sequoia]

Okay, I guess that kind of answers my question, cuz the crux of it is really "how does the deserializer know what it's supposed to be deserializing"

[thorn badger]

For non self describing formats I have the request hint protocol which allows the visitor to "take control" and tell the walker what it should be doing

Similar to serde's deserialize_x methods

[wraith sequoia]

yup yup

[full orbit]

Guys in my async code I have to methods, one populate a DashMap and the other iterate in it and use data from it
The problem is this somehow cause deadlock... Is there any alternative that works In async as well?

[dense oasis]

The dashmap methods return guards. Don't hold those guards across an await point

[modern jetty]

Why is that being asked here lol

[last furnace]

clearly because of their deep appreciation for treaty and a potential replacement for the serde ecosystem

[visual tartan]

How does "Generic types that are generic" have to do with hkt

[dense oasis]

That's what an HKT is

It's a generic generic

HKTs are the ability to use types whose kind is higher than * as generics

A type whose kind is higher than * is a type constructor

A type constructor is an "incomplete" generic type

[visual tartan]

Ah I mean

I have hard time thinking e.g. parametrizing by * -> * as "Generic generic"

Is this in a sense where Foo<F> is generic over F, where F is * -> *, being a generic type itself?

[dense oasis]

Put it like this: Monad m accepts m, which is a generic

However, m accepts a further generic, such as a and b in (a -> m b) -> m a -> m b

Ergo, m is a generic that takes generics

Which we could call a generic generic

[visual tartan]

Yeah that seems roughly what I said

The parameter of the generic type is itself generic, so.. generic of generic. Confusing for me tbh

[dense oasis]

It follows the naming scheme of "generic associated types" 😄

The alternative name would have been "associated type constructors"

[full orbit]

Damn, why did I send it here
That's what sleeping in 11am get you

[full orbit]

Thank you

[wraith sequoia]

what is *

[undone flax]

https://learnyouahaskell.com/making-our-own-types-and-typeclasses#kinds-and-some-type-foo

in the context of rust, you could say i32 is of kind * and Vec is of kind * -> *. note that Vec<i32> will be of kind *, as the type parameter i32 is used to produce the new type

Vec is sometimes called a "type constructor" as it is used to construct different types

[thorn badger]

fn make_list<T: List>() {
  let x: T<i32> = T::from([1, 2, 3]);
}

[undone flax]

Collection trait when

[thorn badger]

it greatly annoys me that fn() gets to be a higher ranked type

for<'a> Closure<&'a i32, &'a i32> I want this

where Closure is a struct

[visual tartan]

How?

[thorn badger]

fn(&i32) -> &i32 is a perfectly fine type

[visual tartan]

I don't see it requiring higher rank.

[thorn badger]

but I can't write this as a struct

struct X<A, B> {
  x: fn(A) -> B
}

its impossible for x to be a fn(&i32) -> &i32

[visual tartan]

Indeed, it's rather an wrapper type for such a function.

Wait

[visual tartan]

Small x?

[thorn badger]

?play ```rs
struct X<A, B> {
x: fn(A) -> B
}

fn main() {
let x: fn(&i32) -> &i32 = |x| x;
let x = X { x };
let x: fn(&i32) -> &i32 = x.x;
}

[tribal rampart]

error[E0308]: mismatched types
 --> src/main.rs:8:31
  |
8 |     let x: fn(&i32) -> &i32 = x.x;
  |            ----------------   ^^^ one type is more general than the other
  |            |
  |            expected due to this
  |
  = note: expected fn pointer `for<'a> fn(&'a _) -> &'a _`
             found fn pointer `fn(&_) -> &_`

For more information about this error, try `rustc --explain E0308`.```

[visual tartan]

Why.

[thorn badger]

poof

its literally impossible to write the type that would allow X to store that function pointer

[visual tartan]

Is this lifetime issue again?

[thorn badger]

no

there is fundamentally no syntax to do it

function pointers and trait objects are the only ones that can use for<'a>

?play ```rs
struct X<A, B> {
x: fn(A) -> B
}

fn main() {
let x: fn(&i32) -> &i32 = |x| x;
let x: for<'a> X<&'a i32, &'a i32> = X { x };
let x: fn(&i32) -> &i32 = x.x;
}

[tribal rampart]

error[E0404]: expected trait, found struct `X`
 --> src/main.rs:7:20
  |
7 |     let x: for<'a> X<&'a i32, &'a i32> = X { x };
  |                    ^^^^^^^^^^^^^^^^^^^ not a trait

error[E0782]: trait objects must include the `dyn` keyword
 --> src/main.rs:7:12
  |
7 |     let x: for<'a> X<&'a i32, &'a i32> = X { x };
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |
help: add `dyn` keyword before this trait
  |
7 |     let x: dyn for<'a> X<&'a i32, &'a i32> = X { x };
  |            +++

Some errors have detailed explanations: E0404, E0782.
For more information about an error, try `rustc --explain E0404`.```

[visual tartan]

But 'a is lifetime, right

[thorn badger]

yes

[visual tartan]

So it sounds like a lifetime issue for me.

[last furnace]

huh

that's dumb

[thorn badger]

for<'a> fn(&'a i32) -> &'a i32 is a higher ranked type

[visual tartan]

Ofc, it's not like you violated lifetime restrictions or such.

[thorn badger]

which I cannot write in my own types

[undone flax]

this reminds that i wish HRTITBs exist

[thorn badger]

pub struct ClosureOnce<Capture, In: Hrt, Out: Hrt> {
    capture: Capture,
    f: for<'a> fn(Capture, WithLt<'a, In>, &'a ()) -> WithLt<'a, Out>,
}

I have to do this

which is pain

[visual tartan]

What does for<'a> mean

[thorn badger]

it introduces a higher order lifetime

[last furnace]

for all lifetimes

[visual tartan]

Ah, higher order lifetime

[thorn badger]

read as "for all lifetimes <type>"

[visual tartan]

That's why this innocent looking fn is somehow higher ranked..

Tho I am confused which one is the rank lifting one

[thorn badger]

any borrow in a function pointer automatically becomes higher ranked

[visual tartan]

Since forall (q :: Lifetime). 'q i32 -> 'q i32 won't be higher ranked

[thorn badger]

isn't for all the literal definition of higher ranked?

[visual tartan]

No

[thorn badger]

every doc I have read has lied to me!

[visual tartan]

https://wiki.haskell.org/Rank-N_types

[thorn badger]

I blame all of the rust docs https://quinedot.github.io/rust-learning/dyn-hr.html https://doc.rust-lang.org/nomicon/hrtb.html

[visual tartan]

(forall a. a -> a -> a) -> (forall b. b -> b -> b) is rank-2 type, since parameter itself is generic with differing arg a.

Ah, maybe this is just different terms then.

Usual programming languages do not have lifetimes, so rust is free to give names regarding lifetimes.

[thorn badger]

I think its better written as (forgive my abuse of notation) forall (q :: Lifetime) 'q -> forall (q' :: Lifetime) where q' < q: 'q' i32 -> 'q' i32

[visual tartan]

Ah, the bottom one example seems higher ranked

[visual tartan]

Yeah, this alone is higher kinded

[thorn badger]

lifetimes have subtyping

[visual tartan]

Where it becomes higher ranked is when you take such a thing as an argument.

Basically, it's about how many independent foralls you can have.

[thorn badger]

ah

yeah everything I have is taken as a parameter

[visual tartan]

Yeah

[wraith sequoia]

@thorn badger what's the ultimate difference between doing like size_of::<&dyn Sized> and size_of::<usize> * 2 when determining the size of fat pointers?

[thorn badger]

Nothing, by the reference they are always identical

At least for trait objects

https://rust-lang.github.io/unsafe-code-guidelines/layout/pointers.html#representation

In treaty I do a sanity check

Just because

I should add a sanity check in the constructor as it takes an arbitrary &T

And technically in the future that could be larger than 2 words

[wraith sequoia]

How does the HigherKinded trait assist in transmuting pointers for AnyTraitObject

[thorn badger]

It provides the type to get a TypeId of

[wraith sequoia]

Okay, I think I have a different problem then, cuz I can't transmute a &'a T to [u8; 16]. It makes sense, but it raises the question of "how do you restrict T to dyn T types" so that the pointers are always fat-sized

[thorn badger]

I don't restrict it to fat pointers (also it's missing the too big check)

[thorn badger]

[MaybeUninit<u8>; 16]

[wraith sequoia]

[MaybeUninit<u8>; 16] or MaybeUninit<[u8; 16]>

ngl if we had Pointee in stable i'd just restrict it to dyn

[thorn badger]

They are both allowed

[wraith sequoia]

So does transmuting a MaybeUninit just make the compiler say fuck it

[thorn badger]

MaybeUninit is allowed to store any bit state

[thorn badger]

And for the actual transmuting I have a special function that makes it easy

https://gitlab.com/konnorandrews/treaty/-/blob/main/src/any/indirect.rs?ref_type=heads#L139

[wraith sequoia]

Isn't that how std does y

It*

[thorn badger]

No, std's transmute has a size check and is a intrinsic

This one allows transmuting from a smaller type to a bigger one if it's MaybeUninit and the other way around

[full hedge]

Consider using Obsidian. I regards its capability to write web-like node files pretty scalable

[full hedge]

What Ortogonal implies? because I only know its usae in geomtery being two lines crossing at 90 degrees

[long sigil]

It means the topic is unrelated

[dawn herald]

In statistics, it means two variates are statistically independent to each other. This definition is what drives the common usage as a synonym for "unrelated"

[wraith sequoia]

@thorn badger have you considered implementing AnyTrait as an enum

And I don't mean AnyTrait particularly, but the concept of it I suppose

Cuz I was thinking it over, and since AnyTrait is basically just a way of generically defining a type union, you could probably shift the syntax load onto an enum instead of with any_trait_impl or whatever the macro was called

[thorn badger]

It would be an enum with an infinite number of variants

In a simplified system without the super flexibility you could

[wraith sequoia]

What part of AnyTrait makes it an infinite type? I think there's a lapse in my understanding. The way it's defined is as a "trait map" afaik, which is inherently finite

[thorn badger]

There are an infinite number of trait objects AnyTrait allows casting to

[wraith sequoia]

But for each thing that implements it there's a finite number, right?

Like per implementation

[thorn badger]

For some of them

Other ones forward their implementation to inner fields

Which is finite but unknown

Also each type having its own enum doesn't really help being generic over it

[wraith sequoia]

Okay, I understand now

My point still stands though, I think you could pull off something like this

#[derive(AnyTrait)]
enum VisitorFamily {
    Value(&dyn Value),
    Borrow(&dyn Borrow),
    #[etc]
    Other,
}

Although I'm not declaring it as the most useful thing ever, it may or may not have benefits

[thorn badger]

So just replace the macro syntax with a enum?

[wraith sequoia]

Yeah p much

[thorn badger]

Hmm, I'm not sure if that would be any less confusing

As for most types each variant is really just &mut self

[wraith sequoia]

I think it's just better to leverage an established syntax than to create a dsl for something that isn't data

[wraith sequoia]

I guess what the user would fill each thing with would depend on where this ends up in their code

[thorn badger]

I have reintegrated the new effects system into treaty, I still have some more work to do on the new system though to reduce the amount of unsafe trait impls treaty has to do

[long sigil]

I read this as

I have resigned from my new effects system

[wraith sequoia]

I gave an implementation a try and saw your point lmao

[thorn badger]

always good to sanity check

[wraith sequoia]

I wish I knew how to like put to words how both approaches feel similar though. I'm sort of struggling to formalize it, but in my head it's kind of like they both represent the same thing in memory, but one is self referential and one isn't? Like functionally, I understand that you're passing the enum's address to itself for that implementation, and AnyTrait basically just does a reinterpret_cast<>(). I feel like it's on the tip of my tongue idk

[wraith sequoia]

@thorn badger Also, there was something else I wanted your opinion on. In my little toy treaty, I had the idea of making an IntoWalker trait that looked like this

trait IntoWalker<W> 
where
    W: for<'ctx> Walker<'ctx>,
{
    fn into_walker(self) -> W;
}

It got me thinking about Visitor as well, because I sort of wonder if it could be implemented the same way. But since there is no equivalent to From<T> for SomeWalker<T> in treaty, I sort of wondered how you thought about it. Technically you could do this, couldn't you? I'm not really sure about costs beyond longer compile times due to the constant monomorphization.

[thorn badger]

I do have this on Walk

[wraith sequoia]

Ik, I'm just talking about hypotheticals

[thorn badger]

im not following what the hypothetical is

[wraith sequoia]

Like, with Walkers, you can construct a walker from a known type, i.e. i32 implements Walk therefore its canonical Walker is ValueWalker or something. With IntoWalker<W>, you're basically requesting if a type can be converted into a walker of a specific ilk at compile time. So, hypothetically, could you implement the same behavior for Visitors? Like instead of it being a runtime request facilitated by AnyTrait and Options, could you do like an IntoVisitor<V> off the expected return type in a Walker

[thorn badger]

if i am following correctly this is basically what serde does

with its accessors

[thorn badger]

I don't really consider this useful as both walkers and visitors need some "special" knowledge to do their work

they can't be themselves generic

I do have StructWalker/StructBuilder and EnumWalker/EnumBuilder that are sort of like this

where the type needs to implement the reflection trait to use them

[thorn badger]

if you swap all the protocol lookups for compile time the bounds explode exponentially

[wraith sequoia]

I appreciate this insight very much :3

Though let me post like a code example just so I know you get me

[thorn badger]

I did try this in a very early version

before this thread was made

[wraith sequoia]

It would basically be something like this

trait Visitor<'ctx> {
    type InOut: ?Sized;

    fn visit(&mut self, value: Self::InOut) -> Self::InOut;
}

trait IntoVisitor<V>
where
    V: for<'ctx> Visitor<'ctx>,
{
    fn into_visitor(self) -> V;
}

trait Walker<'ctx, V> 
where
    V: Visitor<'ctx>,
{
    type Ok: HasVisitor<V>;

    fn walk(self, visitor: impl Visitor<'ctx, InOut = Self::Ok>) -> Result<Self::Ok, ()>;
}

trait IntoWalker<W> 
where
    W: for<'ctx> Walker<'ctx>,
{
    fn into_walker(self) -> W;
}

//

struct NonZeroVisitor;

impl<'ctx> Visitor<'ctx> for NonZeroVisitor {
    type InOut = i32;

    fn visit(&mut self, value: Self::InOut) -> Self::InOut {
        if *self == 0 {
            panic!("go fucking bananas")
        }

        *self
    }
}

impl HasVisitor<NonZeroVisitor> for i32 {
    fn has_visitor() -> NonZeroVisitor {
        NonZeroVisitor
    }
}

struct I32Walker(i32);

impl<'ctx> Walker<'ctx> for I32Walker {
    type Ok = i32;

    fn walk<V: Visitor<'ctx, Self::Ok>>(self) -> Result<Self::Ok, ()> {
        let visitor = i32::has_visitor::<NonZeroVisitor>();
        Ok(visitor.visit(self.0))
    }
}

mod test {
    #[test]
    fn walk_the_walk() {
        let value = 420;

        let walker = value.into_walker::<I32Walker>();
        walker.walk(visitor)
    }
}

sorry if it is actually horrific i got plastered gomen asai -toomah

[thorn badger]

visitors aren't created from values (at least in treaty) they are created from basically a Option<T>

where the value gets inserted into the option

but yeah other than that what you have works, it just doesn't scale

at least i don't know of a way to make it scale

[wraith sequoia]

i think i kind of lost track of what i was writing while i was writing it cuz the visitors arent supposed to be made from values they're just supposed to be gleaned from the type

hold on let m see if i cna edit it to reflect that

lol i started editing it and realized i was just packaging a visitor in the walker but with extra steps

[wraith sequoia]

what would an example of its shortcomings be?

[thorn badger]

to compose these together you need to bound by each field

the complex part of a serialization framework is composition

[wraith sequoia]

sigh one day my apis will be compostable

🪱

[thorn badger]

let mut de = serde_json::Deserializer::from_str("42");
let y = u8::build(DeserializerWalker::new(&mut de));

assert_eq!(y.unwrap(), 42);

See nice and simple API

[last furnace]

gonna post this here so I don't lose it

[digital gazelle]

wait until Treaty has to invent its own acronyms

[lost elk]

I'm pretty sure frog already invented the acronym DIDET

[stone musk]

RPITIT: Ridiculously Pompous Implementations of Traits In Treaty

[digital gazelle]

HBHC: Horrors Beyond Human Comprehension

[mellow thistle]

tbf that's just adding a word onto IDET which comes from gdbstub

[dawn herald]

Me studying all these man-made horrors so they are no longer beyond my comprehension

[sharp condor]

@dawn herald

this u?

[dawn herald]

Mhm

[wraith sequoia]

Tbh I pretty much understand treaty now

[thorn badger]

changes architecture

[wraith sequoia]

Kills you dead where you sit

If I had a website with a blog I'd make a post about it

[modern jetty]

do you have a masto account? that's what I have been doing instead of a blog lol

[wraith sequoia]

Tbh I want to set one up

But there's legit like a demon inside me that whispers "make a custom client" everytime I think abt it

I want to compromise and just like make a website with integration or something

[woven crow]

actually i wonder why does treaty need so much complexity? like isn't a serialization framework just a fn(impl Serialize, impl Serializer) and a fn(impl Deserializer) -> impl Deserialize, maybe with a visitor-thing in between to make user implementation easier/more symmetric

[wraith sequoia]

Tbh to this day I still don't Fully understand why visitors are necessary. But the reason for the complexity is because I don't think the technique can be conveyed in regular Rust

Like even for serde I always sort of questioned it

I could see the convenience but never knew why it wasn't More convenient to just emit the value at deserialization

[dense oasis]

Treaty is a massive pile of complexity to impl TryFrom<A> for B, except A and B do not have to know about each other.

To treaty, serialization and deserialization are the same thing: it's just a conversion between two types

It just so happens that one of those types is trivially representable as a string/byte sequence

[thorn badger]

This is easy to say, but the devil is in the details about how data flows between the two and how you can compose them

It's not much good if you can't compose serializable constructs together and have to manually redo it each time

[thorn badger]

Visitors are the main way to iterate over heterogenous data

Especially if you don't have access to heap to do type erasure

Iterators are themselves actually the most basic form of the visitor pattern if we consider adapters

[thorn badger]

Also a large amount of the complexity of treaty is my want to be generic over async and blocking code

While still being no_std and near perfect codegen

The core DIDET isn't actually that complex

The only weirdness is the fatter pointers

[wraith sequoia]

Yeah it's pretty much that

[modern bone]

Watch the frog as they calmly tell lies
cries in closures

[thorn badger]

Be gone chicken

[wraith sequoia]

How do you plan to solve the boilerplate around AnyTrait

[modern bone]

[wraith sequoia]

https://tenor.com/view/deer-snow-interview-microphone-mic-gif-25816123

[thorn badger]

I don't currently have any plans, as it's in the internals that only format integrators would use. We can readdress it after the alpha

[wraith sequoia]

I was thinking about it and really I can only think of like supertraits of AnyTrait

But we can talk l8r

[thorn badger]

I'm sure the alpha is going to be "frog feature X is missing/broken "

I'm interested in the "will people actually care to use such an API"

[wraith sequoia]

Me too tbh

I feel like there's a possibility people may shy away because of dark magic, but at the same time who tf actually learns the internals of the apis they consume

[dawn herald]

Raises paw

[wraith sequoia]

I mean yk me too

[forest osprey]

One thing I was wondering is, could this be used for AST transformations?
E.g., If I have an Ast<Parsed> and I want to convert it to an Ast<Interned>, that would be a lot of boilerplate to write manually. But maybe serializing it from one to the other could be nice?

eh maybe not. But I have been thinking about this

[thorn badger]

Would definitely be an interesting usecase to try out

Treaty is really good at that kind of "keep the overall structure but mutate some of the inside stuff"

All my testing so far has actually been for struct to struct/enum to enum

As those can have their implementations auto generated

Which is one usecase I haven't really seen any serde code attempt outside of maybe using serde_json::Value

[dense oasis]

Neat, it's a hylomorphism

For once, the terminology of recursion schemes sounds simpler than the alternative. I didn't think this was possible 😄

A hylomorphism, of course, is just a catamorphism combined with an anamorphism

(translation: it's fold combined with unfold, generalized to arbitrary recursive data types. It keeps the shape and replaces everything in it)

[thorn badger]

Yeah it's more or less exactly that

[dense oasis]

The precise intuition here is that you're merging the steps of "combine everything into a single intermediate value" and "blow up that intermediate value into another data structure"

It's actually kind of neat to see it in practice

[thorn badger]

Where here we don't have a set intermediate

You define a fold or unfold operation and it can be combined with any of the opposite form

At some point I am going to need to make some micro walkers and builders with the best codegen for specific types for things like embedded. As the ones that are super compatible cause llvm to die trying to optimize them away

No need to change the core API of treaty though

Just which ones you use

[thorn badger]

Can we just consider treaty as an attempt to apply functional programming theory into something actually useful

All I really want out of treaty is to prove something better than serde can exist and to at least motivate the development of those systems

[wraith sequoia]

If I ever meet you I should game end you with a rock and eat your brain for its infinite wisdom

[wraith sequoia]

It honestly makes me so fucking happy that learning about really advanced Rust has forced me to learn functional concepts

[dense oasis]

I am going to catamorphise (fold) you first

[wraith sequoia]

You can catamorphise this weenie

https://tenor.com/view/basketball-slam-dunk-man-dunks-basketball-basketball-hoop-gif-26961879

Seriously though I need to like pick up Haskell and carry over a bunch of stuff to Rust

[dense oasis]

If you want the Cooler Haskell, pick up Idris 2, with its dependent types. Haskell itself is easier to rust-represent, though

[wraith sequoia]

Strokes chin

I'll look at em

[fair acorn]

me when i #[repr(Rust)] in haskell

[nimble dawn]

what is the best place to start if I want to figure out how treaty works

[undone flax]

learn haskell

[nimble dawn]

lol

[lost elk]

get a phd

[dense oasis]

For serious answers, get good at traits

Then, ask Frog how DIDETs work, I don't think we have a proper guide anywhere

[undone flax]

tbf treaty is somewhat readable when you are used with typenum

lol

[nimble dawn]

what does DIDET stand for

[dense oasis]

Why yes, climbing the K2 is easy, as long as you climb the everest frequently

[lost elk]

dynamic inlineable dyn extension traits I think

I don't remember what the first D stands for

[undone flax]

[lost elk]

but IDETs are not a treaty invention, they're described here https://github.com/daniel5151/inlinable-dyn-extension-traits

and here https://docs.rs/gdbstub/latest/gdbstub/target/ext/index.html

[undone flax]

im not keen on treaty yet, is that like dungeon-cell tech

[wraith sequoia]

If you're familiar with C++'s reinterpret_cast<>() it basically functions similar. A core idea is that anything implementing AnyTrait can have its dyn pointer cast to any other dyn pointer, therefore letting it become "any trait" (within the bounds of a predefined typemap). It's literally implemented as like match typeid { supported_type => AnyTraitObject::new(&mut self) } idk if that's precise cuz I'm on my phone but yeah

[thorn badger]

I definitely didn't make reflection at home

[wraith sequoia]

Otherwise, Walkers model the structure, they request functionality from an AnyTrait visitor, and can decide what to do based on its reply, because it returns Options instead of panicking

Then the visitor functionality they receive does stuff with the data and then you get your value

[thorn badger]

See simple

[wraith sequoia]

Might be kind of oversimplified but that's the gist AFAIK @nimble dawn

[thorn badger]

Senor is my first follower

[nimble dawn]

could you point me to the code that does that type lookup?

[wraith sequoia]

It's in any.rs

[nimble dawn]

thanks

[wraith sequoia]

My name is toomah...

[thorn badger]

Senor is senor

[wraith sequoia]

Blahhh

[thorn badger]

Fatter pointers go brrrrr

[wraith sequoia]

But yeah most of the complexity comes from frog being a schmuck and adding effects and shit

[thorn badger]

I am weird what can I say

[nimble dawn]

I assume you could do something similar to treaty but without async support and only owned/Clone values and it would be a lot simpler?

[wraith sequoia]

Yes

[nimble dawn]

I kind of want to make that, baby's first treaty

[thorn badger]

do it

[nimble dawn]

if I do I will write some blog posts about treaty internals

and use writing baby's first treaty as an introduction (for me and the posts)

[nimble dawn]

question, the current treaty from gitlab doesn't compile on my machine, it says it's missing the effectful module. Is that intentional? or am I being silly and missing something

[thorn badger]

I haven't made the effectful crate public yet

[nimble dawn]

ah I see

unfortunate, I was hoping to get type inlays

[thorn badger]

https://gitlab.com/konnorandrews/effectful

should be visible now

[nimble dawn]

many thanks

[thorn badger]

I think I left it in a buildable state

I usually commit at the end of the day after working on it

[nimble dawn]

even if it doesnt quite build I should still hopefully get type inlays

[thorn badger]

so no guarantee it actually builds at that point

[nimble dawn]

it builds anyways, nice

[thorn badger]

to be honest the type hints may not be that helpful

effectful messes with them

[nimble dawn]

I see

at the very least I can write code that pokes it and actually run that code

[nimble dawn]

I see what you mean

[thorn badger]

yeah

i have tried a few things to convince it to not do that

but rust analyzer ignores things like type aliases

[nimble dawn]

question, I'm looking at NoopVisitor, and reading the AnyTrait comments

any_trait! {
    impl['ctx][E] NoopVisitor = [] where E: Environment
}

what does this end up doing?

since the list of traits to register for NoopVisitor is empty

[thorn badger]

it means NoopVisitor always returns None

[nimble dawn]

Ah

[thorn badger]

as it's list is empty

aka it implements no dynamic traits

[nimble dawn]

because it doesn't implement any traits that would allow a walker to get a value from it?

[thorn badger]

No-op is a visitor (or walker) that doesn't actually do anything

[thorn badger]

the walker would want a trait to inject a value, but yes

data flow is usually walker to visitor

[nimble dawn]

can you explain what you mean when you say the trait injects a value?

[thorn badger]

https://gitlab.com/konnorandrews/treaty/-/blob/main/src/protocol/visitor/value.rs?ref_type=heads#L36 is the prime example, visitor.visit(42) would inject the value 42 into the visitor

[modern bone]

first victim

[wraith sequoia]

@thorn badger is the bijectivity test pivotal to AnyTraitObject::into_inner()'s soundness

[thorn badger]

yes

otherwise the same TypeId could be used for two different types

[wraith sequoia]

like two types might have clashing ids or because RawIndirect can be cast to anything

[thorn badger]

going back to normal type ids, say TypeId::of::<A>() and TypeId::of::<B>() have the same answer but A and B had different layouts and fields and what not. Any would allow A and B to transmute between them, which would be very bad

[nimble dawn]

can someone explain how AnyTraitObject works?

my understanding is that's a wide (trait object) pointer but with the type as a runtime value returned by the type_id function

Oh

no I still dont get it lol

[thorn badger]

so a &dyn Trait stores a pointer and the vtable for Trait of the value, right?

[nimble dawn]

right

[thorn badger]

AnyTraitObject adds a second vtable

so pointer + vtable for the trait + vtable for AnyTrait

[nimble dawn]

Ohh

so it's like a dyn AnyTrait

higher ranked trait object sorta deal

[thorn badger]

this is because we can't put AnyTrait's vtable info into the vtable for the trait as a super trait would

[thorn badger]

yeah

the names need some work

[nimble dawn]

right because AnyTrait doesn't know about all of its extensions (wheras in IDETs the main trait is explictly aware of all of the extensions)

[thorn badger]

im probably going to invert it so its something like WideRef<'a, dyn AnyTrait>

[thorn badger]

yep

hence my dynamic prefix

[nimble dawn]

ok DIDETs make sense now

[thorn badger]

instead of a fixed list defined by a trait we allow it to be dynamic

[nimble dawn]

I still don't quite get the trait object upcasting, or precisely how the walkers/visitors actually exchange information

[thorn badger]

yeah the protocols (the traits) and flows (the abstract sequence of trait uses) has taken me a while to work out

[nimble dawn]

I can see why you described DIDETs as reflection at home lol

[thorn badger]

yeah, the fun part is llvm can see through it a lot of the time so you don't have the cost of a runtime reflection lookup

[nimble dawn]

yeah, hence the inlineable

"hello there yes mr big blue c++ dragon please see through my indirections please :3"

[thorn badger]

in reality treaty is kind of a big reflection crate at different levels of Rust from values to types to traits

which makes sense as a serialization framework is kind of by definition a reflection framework

[nimble dawn]

good point

[thorn badger]

if you are interested I recently finished reading https://recursion.wtf/posts/rust_schemes/ for a different flavor of this type of thing

I don't think it has direct application for treaty

[nimble dawn]

what is the difference between the of, of_lower and of_value methods on TypeNameId

If I had to guess I'd say it has something to do with treaty needing to be generic over borrow lifetimes but I'm not entirely sure

or I guess my question is what is meant by the doc comments when they mention "lower type"

this is probably me being hindered by my lack of proper knowledge of type theory stuff

[thorn badger]

in this context lower means a type like A<'a, 'ctx> where we have lifetimes, and higher means a type like A without lifetimes

treaty defines a mapping between them

[nimble dawn]

so a lower type is the higher type with a lifetime "filled in"?

[thorn badger]

yeah

[nimble dawn]

so 'a is the lifetime of the borrow to be filled in, I've seen 'ctx a lot, what does it usually represent?

[thorn badger]

'ctx is a lifetime that kind of infects everything, in serde land it's the 'de lifetime, it represents the lifetime of the source data

[nimble dawn]

Ahhh

thanks, that helps a lot

[thorn badger]

its there to allow zero-copy stuff where you can just borrow the source data

[modern bone]

Huzzah, now I can embed JSON directly into my rust program, and parse it in a Lazy, without copying!

[nimble dawn]

so a higher ranked (unlifetimed type) has a different TypeNameId than its various lower-ranked lifetime'd derivatives?

[thorn badger]

Rust currently doesn't allow non-'static types to have type ids

[nimble dawn]

Oh

so that's why of_lower is its own method

[thorn badger]

so to get some for them we have to remove the lifetimes

[thorn badger]

yeah it raises the type to the non-lifetime containing form then takes the TypeId of that

[nimble dawn]

Ok this makes more sense now

it takes the lifetime out of the given lower type so the it has something 'static it can get the type id of

[thorn badger]

yep

the complexity is in proving that is sound to do

which yandros helped me with

originally for dungeon-cell

[nimble dawn]

I'm trying to reason about where the soundness issue for that is

[wraith sequoia]

impl AwesomeId {
    fn of_lower<T>() -> TypeId {
        struct Phantom;

        return TypeId::of::<Phantom>();
    }
}

[thorn badger]

its when we transmute the values based on the type id being equal

[nimble dawn]

yeah I saw the comment in any.rs

[thorn badger]

if only

[wraith sequoia]

does that not work

[thorn badger]

you only get one Phantom type

its like statics

[wraith sequoia]

struct Phantom<M>(PhantomData<M>);

https://tenor.com/view/in-all-gif-23325011

[thorn badger]

if M is not 'static then Phantom isn't

[wraith sequoia]

had a feeling that's what u would say

[modern bone]

?eval

use std::any::TypeId;
fn f<T>(_: T) -> TypeId {
    struct Phantom;
    TypeId::of::<Phantom>()
}

(f(true), f("false"))

[tribal rampart]

(TypeId(0x40ae18a651d35763227a974fbb74f6c6), TypeId(0x40ae18a651d35763227a974fbb74f6c6))```

[nimble dawn]

same thing applies to closures

[modern bone]

Is there a way to embed a type with a lifetime into a struct without affecting its lifetime?

[thorn badger]

nope

[nimble dawn]

Not without unsafe (plus somehow proving soundess of whatever you did)

[thorn badger]

if it was possible you could use Any to commit UB

[modern bone]

Well unsafe is "fine" here considering we're not actually using the type

[nimble dawn]

yeah but in order to do the embedding you have to "forget" the lifetime

[modern bone]

I'd take the "it's probably fine, ship it" route, and leave it to the horrified future maintainers to fix 5 years later when treaty is popular

[nimble dawn]

you'd just have

struct RefNoLifetimeUnsafe<T>
  ref_to_t: *const T
}

[thorn badger]

also doesn't work

[nimble dawn]

yeah

[thorn badger]

you need to use dungeon-cell style type erasure

[modern bone]

Which seems no less cursed than what you're doing

[nimble dawn]

I think that's what kyllingene was getting at

[modern bone]

Or is it what you're doing? I know you mentioned dungeon-cell earlier

[thorn badger]

it doesn't help me get a unique TypeId

all ```rs
struct NoLifetime {
magic: Hidden<T>,
}

have the same TypeId

[nimble dawn]

from treaty

impl TypeNameId {
    ...
    /// Get the type ID from a lower type.
    #[inline(always)]
    pub fn of_lower<'a, 'ctx: 'a, T: ?Sized + TypeName::LowerType<'a, 'ctx, E>, E: EnvConfig>(
    ) -> Self {
        Self {
            name_id: TypeId::of::<TypeName::HigherRanked<'a, 'ctx, T, E>>(),

            #[cfg(feature = "better_errors")]
            name: type_name::<T>(),
        }
    }
  ...

[wraith sequoia]

how often do you spend dicking around to learn these things

[nimble dawn]

why does HigherRanked here still need 'a passed in as a type param?

[thorn badger]

like every afternoon

[thorn badger]

because of how I prove the bijectivity in the type system

if you use unsafe instead then you don't need them there

[nimble dawn]

I see

and that has to do with the bijectivity proof function in any.rs

[wraith sequoia]

why?

[thorn badger]

I don't use unsafe to just promise the types are bijective, I actually have rustc prove it, though I may change that do to the extra load on rustc

[thorn badger]

makes every implementation safe to do

impl<'a, 'ctx, T: ?Sized, E> TypeName::MemberTypeForLt<'a, 'ctx, E, &'a &'ctx ()>
    for ValueProto<T, E>
where
    E: Environment,
    T: TypeName::MemberType<E>,
{
    type T = dyn Value<'ctx, T, E> + 'a;
}

impl<'a, 'ctx, T: ?Sized, E> TypeName::LowerTypeWithBound<'a, 'ctx, E, &'a &'ctx ()>
    for dyn Value<'ctx, T, E> + 'a
where
    E: Environment,
    T: TypeName::MemberType<E>,
{
    type Higher = ValueProto<T, E>;
}

for example

no unsafe here

and yet we know that dyn Value<'ctx, T, E> + 'a and ValueProto<T, E> are bijective

and no other types could be using them

[wraith sequoia]

ok so i think im actually picking up what youre putting down

[modern bone]

Put it back, I set it there for a reason

[wraith sequoia]

but i think in this moment i must ask an integral question

what actually constitutes "higher" and "lower" in this Type Terminology

[nimble dawn]

lifetimes

[thorn badger]

higher - no lifetimes ('static), lower - lifetime containing

[nimble dawn]

for<'a> &'a T is higher that &'a T

[wraith sequoia]

https://tenor.com/view/kiss-gif-11816971814746635421

[thorn badger]

its as if type A<'a> = ... could just be A and be a type

(home made type constructors)

[nimble dawn]

the higher ranked lifetimes rabbit hole goes deep

[thorn badger]

?play ```rs
type A<'a> = &'a str;

dbg!(core::any::TypeId::of::<A>());

[tribal rampart]

[src/main.rs:4:1] core::any::TypeId::of::<A>() = TypeId(0xb98b1b7157a6417863eb502cd6cb5d6d)```

[modern bone]

When will ferris finally end our sufferings and ease the pain of these behemoth constructions

[thorn badger]

rustc you aren't helping

[nimble dawn]

lmao

what just happened lol

oh

[thorn badger]

it forced it to be A<'static>

[nimble dawn]

its getting inferred to 'static

yeah

I love static lifetime elision

[thorn badger]

?play ```rs
type A<'a> = &'a str;

struct X {
x: A,
}

[tribal rampart]

error[E0106]: missing lifetime specifier
 --> src/main.rs:5:6
  |
5 |   x: A,
  |      ^ expected named lifetime parameter
  |
help: consider introducing a named lifetime parameter
  |
4 ~ struct X<'a> {
5 ~   x: A<'a>,
  |

For more information about this error, try `rustc --explain E0106`.```

[thorn badger]

well at least that one works

[modern bone]

What would it mean for a struct to contain a type constructor?

Oh oh oh how about type-level structs

[thorn badger]

X would become a type constructor (assuming i remember my type theory)

[modern bone]

Ah

I believe you, considering you've been living and breathing this stuff recently (even if not that particular bit)

[thorn badger]

or a dependent type

[wraith sequoia]

So Like... da lower's associated types kind of "hang onto" the lifetimes for the higher, and they can convert between each other?

[nimble dawn]

opposite I think?

[thorn badger]

the traits sort of link the lifetimes together even though the associated types don't have them directly

[wraith sequoia]

through 'a and 'ctx yas

[thorn badger]

yeah, ValueProto<T, E> from above doesn't have the lifetimes, but the trait impl of TypeName::MemberTypeForLt connects it to the one that does

in such a way that you can't escape

[nimble dawn]

that is clever

[thorn badger]

if you attempt to break out rustc doesn't let you with either a conflicting impl or a missing impl

[nimble dawn]

so what does MemberType represent?

[thorn badger]

this whole idea was yandros'

[nimble dawn]

I haven't talked to yandros but I have heard multiple people mention that they are a Cursed Rust Master

[thorn badger]

that we can lower from Self to a lifetime containing type by giving the lifetimes, meaning Self is a member of a higher ranked type class

[modern bone]

There's a reason :ferrisCursed: is yandros' pfp

[nimble dawn]

and MemberTypeForLt is the result of the lowering, with a concrete lifetime 'a?

[thorn badger]

opposite

[thorn badger]

we get the lowered form (the real type) with <Higher as MemberTypeForLt<'a, 'ctx, E, &'a &'ctx ()>>::T;

Higher + 'a + 'ctx = Lower (T)

[nimble dawn]

Oh

so it performs the lowering via associated type

[thorn badger]

yep

We raise by doing <Lower as LowerTypeWithBound<'a, 'ctx, E, &'a &'ctx ()>>::Higher

which is like Lower - 'a - 'ctx = Higher

[nimble dawn]

lifetime erasure

is how I want to describe this

[thorn badger]

the bijective test is if (Lower - 'a - 'ctx) + 'a + 'ctx = Lower (and the opposite direction)

[wraith sequoia]

we should come up with fun names for these traits

i must go to bed. Goondight

[nimble dawn]

gn

[thorn badger]

yep its a good name for it

[nimble dawn]

this is so cool lol

[wraith sequoia]

lifetime reanimation

[nimble dawn]

I'm glad I decided to dig into this

[thorn badger]

so while we have been discussing this I have been trying to update AnyTraitObject to be simpler and I realized something

it actually should have 3 lifetimes

&'a mut dyn Trait<'ctx> + 'lt is covariant over 'a but invariant over 'ctx and 'lt

[nimble dawn]

that makes sense

[thorn badger]

but I currently have ``&'a mut dyn Trait<'ctx> + 'a`

which is probably the source of some lifetime issues I had previously

[nimble dawn]

sorry I'm a little rusty on lifetime bounds, what does the + 'lt represent about the type again?

[thorn badger]

means the trait object can live for 'lt

[nimble dawn]

Ah

[thorn badger]

aka it contains no lifetimes smaller than 'lt

[nimble dawn]

what differentiates that from 'ctx?

[thorn badger]

'ctx acts as the max

in other parts of the code

[nimble dawn]

Oh

[thorn badger]

'lt will always end before the walking is complete

'ctx wont

[nimble dawn]

so the trait object is borrowed for 'a, borrows dependencies for 'ctx and is alive for 'lt?

[thorn badger]

yeah

it gets even more confusing in some parts where we get more lifetimes that fit into this

[nimble dawn]

so the cases that caused issues with only 2 lifetimes were where the trait object needed to outlive the lifetime it was borrowed for

[thorn badger]

it means rustc locked the borrow lifetime to what I had on the trait object

like &'static mut dyn Trait + 'static

not a very helpful thing

[nimble dawn]

ah

[thorn badger]

If you really want your brain to break you should dig into implied lifetime bounds

those hurt me

[nimble dawn]

in your testing, did LLVM inline through AnyTraitObject?

[thorn badger]

yep

[nimble dawn]

that is... kind of astonishing

[thorn badger]

though there are some cases where it breaks the inlining

and it gives up

my current knowledge is llvm is somehow running out of virtual registers during the inlining

who thats possible i have no idea

considering they are virtual

[nimble dawn]

considering the MaybeUninit casting shenanigans in RawIndirect

[thorn badger]

yeah its quite cool

I gave it a 50/50 chance of working originally

[nimble dawn]

I thought the whole point of virtual registers is that you aren't supposed to be able to run out of them lol

maybe there's a per-opt pass budget or something?

[thorn badger]

you know, i would have that that also

but when I check the reason the pass gives for failing thats what it says

[nimble dawn]

maybe it has a limit to keep the pass runtime reasonable

[thorn badger]

yeah its probably something like that

even if it doesn't inline properly the performance should be good as the branch predictor can take over

[nimble dawn]

yeah, even without inlining it's still probably really competitive performance wise for what it can do

DIDET/IDET is a neat pattern

[thorn badger]

its funny that I found the gdbstub crate randomly

because i wanted to debug some unrelated arduino code I had

https://github.com/daniel5151/gdbstub/issues/140 I actually broke llvm trying that

[nimble dawn]

that's funny

[nimble dawn]

another question, what is the Available associated type on AnyTrait for?

[thorn badger]

documentation purposes, im going to remove it

[nimble dawn]

I see, I had an inkling that it did nothing because doing anything practical types-wise with a tuple like that would be a gigantic pain

also, why is your INDIRECT_SIZE const multiplying size of usize by two, not

const WIDE_POINTER_SIZE: usize = core::mem::size_of::<&dyn Any>();

[thorn badger]

no particular reason

[nimble dawn]

I see

I am starting on entente (baby treaty)

An informal alliance or friendly understanding between two states.

[thorn badger]

a lot of specifics are just "because thats what worked when I was messing with it"

[nimble dawn]

I feel like the difference in definition will accurately convey the differences between the two lol

where 'a: 'b means where a outlives b right

I always get this mixed up

[thorn badger]

yes

[modern bone]

I think my own lib might(!) be coming along, though it's getting a little more complicated than I'd hoped

[nimble dawn]

wait this is cursed

is the union transmute thing in RawIndirect needed for llvm to optimize though anytraitobject

also, why is the info field of AnyTraitObject a function that returns a TypeNameId instead of just storing the TypeNameId directly

the comment implies that it was a vtable with a drop function at one point, so maybe that's why?

[thorn badger]

no, but the normal transmute gets upset do to it being a generic

[thorn badger]

yes, but also a TypeId is now larger than a function pointer, so i use it to save space

[nimble dawn]

i see

couldn't you just use &'static TypeNameId

and const

[nimble dawn]

is this because it can't confirm both types' sizes

[thorn badger]

ish

while https://doc.rust-lang.org/nightly/std/any/fn.type_name.html isn't const I can't static promote the value when its storing the type name

[nimble dawn]

oh type_name isnt const??

[thorn badger]

yeah

[thorn mantle]

https://github.com/rust-lang/rust/issues/63084

[nimble dawn]

funny

[wraith sequoia]

It confuses the shit out of me why TypeId isn't for any lifetime and type_name() isn't const

[dense oasis]

The first one is, as it turns out, required for soundness

The lifetime system requires parametricity, aka the property that a generic function cannot know anything about its generic parameters except for what its bounds say. Put another way, if there was any way to ask "are these two actually the same lifetime" or "is this 'static, by any chance" or "does this live longer than that" that would be unsound

The second one is because we have not implemented it yet

[wraith sequoia]

Honestly I think a lot of people would appreciate a function that could return an ID that's affected by the lifetime bounds, but I know that's likely not feasible due to their current nature

And I mean something unique across T or T + 'a etc

[long sigil]

Unfortunately rust never knows the lifetimes, it just asserts the requirements are met

[wraith sequoia]

Exactly

[visual tartan]

Rust never.. huh?

[thorn mantle]

lifetimes are erased pretty quickly

they're checked then discarded

[visual tartan]

Ah, that's sad

It would be great imo to be able to inspect lifetimes.

[thorn mantle]

if im understanding t-dark correctly that's unsound but i couldnt tell you why

[long sigil]

Should I use treaty or should I use https://medium.com/@abdulgh/compile-time-json-deserialization-in-c-1e3d41a73628

[thorn badger]

I want to see you use template heavy C++ in Rust

[long sigil]

Just watch me

[thorn badger]

impl<'ctx, T: 'static, E: Environment> AnyTrait<'ctx> for ValueBuilder<T, NotCloneable, E>
{
    fn upcast_by_id_mut<'a, 'lt: 'a>(
        &'a mut self,
        id: WithLtTypeId<'lt, 'ctx>,
    ) -> Option<MutAnyUnsized<'a, 'lt, 'ctx>>
    where
        'ctx: 'lt,
    {
        trait_by_id!(&mut self, id, {
            type Impls = (dyn RequestHint<'ctx, E>, dyn Value<'ctx, OwnedStatic<T>, E>);
        });

        None
    }
}

new syntax drop

[modern bone]

Does treaty have more collective syntax than Rust yet?

[visual tartan]

Btw how about making your own lang

( )

[next tundra]

Yeah you might be due for a DSL at this point

[wraith sequoia]

Ok like consider the following

Small thing that generates full serde-like structures...

This has nothing to do with treaty I'm just saying this here cuz this is serialization central now

[modern bone]

You mean, something like a parser generator, that makes a serde-style non-flexible ser/de framework but tailored to your needs?

[wraith sequoia]

Yeah pretty much

Don't tell me; it already exists

[modern bone]

Doubt it

I doubt this'd be that useful in the end, but idk

[wraith sequoia]

If I can make something simple I think it would be useful

Only because I wouldn't be wasting all of my time writing it

I think I might just do something kind of rigid for my own purposes

Rubs my disgustingly crusted greasy hands together

[bronze dune]

Hello, i am finally free enough to dirty my hands with treaty, but I am confused on which file to start. Any recommendations? I dont see anything that could explain the mental mode behind treaty.

[digital gazelle]

the best i've found is just reading the chat logs here

[thorn badger]

I would recommend starting with the explanation at https://docs.rs/gdbstub/latest/gdbstub/target/ext/index.html as that was the source inspiration for the technique

[bronze dune]

So if I understand this correctly(i likely dont), if something wants to serialize something, it provides a visitor, and each visitor implements certain "protocols" which a walker can check and feed values accordingly..? and the complex stuff comes from effects and non-'static any?

[thorn badger]

Yeah

[bronze dune]

what file/trait/whatever is the root of your "async vs sync" stuff? the effect system? how does it work?

[thorn badger]

thats now in a separate crate

https://gitlab.com/konnorandrews/effectful

[bronze dune]

which would be?

thanks, this will be really helpful

[thorn badger]

instead of looking at main look at https://gitlab.com/konnorandrews/effectful/-/blob/9c15ab384cbb262fd6d5dafcbafc5706dd6245ef/src/effective.rs

im current refactoring it

[bronze dune]

reason of death: they saw the bounds on helper methods

[wraith sequoia]

@thorn badger honestly the refactor makes this stuff way easier to read

but i feel like i want to ask, How Does the Raising-Lowering Actually Work?

[thorn badger]

yeah but currently its missing most of the API

[thorn badger]

associated types

to implement Type -> Type functions

[wraith sequoia]

hmmmmmmmmmmhowdoyoumean

[thorn badger]

trait Unsign {
  type Type;
}

impl Unsign for i32 {
  type Type = u32;
}

type ToUnsigned<T> = <T as Unsign>::Type;

here ToUnsigned forms a function f(i) -> u where i is a signed type and u is a unsigned type

the lowering and raising are two of these r(L) -> R the raising function, and l(R) -> L the lowering function

when r(l(R)) -> R and l(r(L)) -> L

and R: 'static

type level programming go brrr

[wraith sequoia]

https://tenor.com/view/aaa-gif-27514809

[next tundra]

https://tenor.com/view/yoink-cat-dragged-gif-1263419341748157821

[thorn badger]

https://gitlab.com/konnorandrews/effectful/ effectful is almost put back together

[long sigil]

[modern bone]

Time until nuclear impact: 3 days

[south terrace]

when can i expect my fight of the year movie with effectful fighting against effective?

[tiny geode]

This looks wonderful but also like if I tried to use it,

rustc would use 37 GB of memory and 15 minutes of CPU time to physically manifest Ferris to stab me

[undone flax]

ferris would be too busy typechecking the program to do that

[digital gazelle]

--rustflags=skip-typechecks

[modern bone]

That's just called mrustc

[sharp condor]

[thorn badger]

if anyone feels like attempting to solve a hard lifetime issue, https://gitlab.com/konnorandrews/effectful main branch (with effectful's main branch) has a lifetime error I can't track down the cause of

[rustic glade]

if u can't find it...

[nimble dawn]

[wraith sequoia]

So I was thinking about it, and is the use of all the raising and lowering of types just to extract their TypeIds?

[thorn badger]

Mostly

[wraith sequoia]

So what if there was just a Different Way

[thorn badger]

treaty API refactor v5

[next tundra]

Hell yeah

[full orbit]

Are we close to beta yet?

[modern jetty]

every refactor is another 1.5 months, hence it being 7 months since announced

[open oriole]

Are we even in alpha yet

[long sigil]

Are we even

[austere saddle]

Are we

[digital gazelle]

-#

[grizzled bison]

[thorn badger]

We should have something before alpha

[dawn herald]

I know!

We'll call it pre-alpha

[austere saddle]

delta when

[next tundra]

Let's call it prod

[wraith sequoia]

sigma

[visual tartan]

is it out yet

[visual tartan]

Hmmm

[wraith sequoia]

https://tenor.com/view/im-out-bye-gone-dead-goodbye-gif-22430671

[sharp condor]

https://tenor.com/view/komaru-comaru-komaru-cat-komaru-cat-explosion-cat-explode-gif-1725161613750857467

[restive ravine]

is there an example of a minimal usage of treaty?

the tests in the repo are all commented

[thorn badger]

I am kind of in the middle of fixing it after the effectful refactor

[long sigil]

... Usage?

[restive ravine]

wait what did i say

i'm basically just asking about the real simple serialization deserialization sorta cases, nothing particularly fancy

just curious what the very general API surface looks like

[thorn badger]

https://gitlab.com/konnorandrews/treaty/-/blob/main/tests/serde_deserializer.rs?ref_type=heads is what the normal user would see

Aka a user that isn't implementing a format

It's basically identical to serde

[restive ravine]

oh okay

[long sigil]

No you said it correctly, but treaty has a use?

[open oriole]

is there a usable ver yet :p

[thorn badger]

Not yet

I have gone a side tangent while making the proc macros

[thorn mantle]

i am curious, does treaty support lazy deserialization?

[thorn badger]

as in?

[thorn mantle]

something like serde's RawValue

[thorn badger]

oh yeah it can totally do that

[thorn mantle]

where some inner value has an unknown type at the time of deserialization

huge

[thorn badger]

in treaty you can just give the input to the builder

and the builder can put it as is into the output

[thorn mantle]

i assume that wouldn't work for something like yaml

[thorn badger]

it works for anything

on a side note treaty also allows things like yaml's references

which means it can construct cyclic values

[full orbit]

Wait a minute, what's Build! in derive macro? Is it just a macro? Didn't know we can do this in derive macro...

[thorn badger]

You can thank yandros for that

[full orbit]

How exactly does it work tho? I mean why not just use proc macro directly here?

[thorn badger]

I don't want to bother with a proc macro for feasibility testing

[full orbit]

I'm more impressed that we even can use decl macro like this... I mean I never seen anyone use it like this...

I can't even find anything explaining how does it work...

[thorn mantle]

there's a crate that provides its own #[derive]

#[derive(Ident!)] item expands to Ident! { item }

[full orbit]

Oh... So it's not something that default derive macro allow...
Now it make sense...
I didn't saw that derive is imported from macro_rules_attribute...

[thorn badger]

Yeah a yandros crate, hence my original comment

[pearl zephyr]

Frog, is your crate going to support full zero copy deserialization?

I know that serde already supports this to some degree, but it's not exactly ... pleasant to work with

[thorn badger]

yes, though its unlikely to be as much as rkyv

though i am planning on attempting to wrap rkyv's api in treaty's

[pearl zephyr]

Yayy, that still makes me happy. I'm pretty sure it'll be better than serde; that's what matters most

[wraith sequoia]

I wanted to ask since I think I'm trying to write something similar conceptually, does a Walker type always know what protocols it's going to request from a DynVisitor?

[thorn badger]

Usually it's static but it can be decided at runtime

[wraith sequoia]

But in like, either have just one definite call (i.e. visit_value(...) and nothing else, being static) or like a handful of if else (technically runtime)

[thorn badger]

Currently how I have it setup walkers will attempt the RequestHint protocol, if that doesn't work they will then emit any number of tag protocols and one of the main protocols (value or sequence)

This is not required to do though

[wraith sequoia]

right ok

[forest osprey]

Today I once again considered treaty for writing compiler passes

specifically for dealing with tree-sitters inconvenient cursor api, which probably lends itself quite well to being used by treaty

[open oriole]

:3

[wraith sequoia]

Okay, hear me out

I've been workshopping with simpler ideas and now instead of Walkers I'm calling them Guests cuz they facilitate a visit Lol

I think what I have so far

pub trait Visit<'ctx, T: ?Sized, E> {
    type Value;

    fn act(self, value: T) -> Result<Self::Value, E>;
}

pub trait Guest<'ctx> {
    type Value;
    type Error;

    fn visit<V>(self, visitor: V) -> Result<Self::Value, Self::Error>
    where V: Visit<'ctx, Self, Self::Error>;
}

Is still too basic to be versatile enough though

I really want to do this

impl<'ctx, T: ?Sized, U, E, F> Visit<T, E> for F
where
    F: FnOnce(T) -> Result<U, E>,
{
    type Value = U;

    fn act(self, value: T) -> Result<U, E> {
        (self)(value)
    }
}

impl<T> Visit<T> for T {
    type Value = T;

    fn act(self, value: T) -> Result<U, E> {
        Ok(value)
    }
}

But no specialization 👎

[forest osprey]

what's the difference between a guest and a visitor? I'm not convinced by the terminology :P

[wraith sequoia]

None, the Guest replaces the Walker terminology

[forest osprey]

oh visitor is a visit

that's even more confusing

i think it should be guest.visit(host)

just because that's more fun

[wraith sequoia]

Yeah it should Lol

But the whole reason I switched to Visit instead of Visitor is cuz the Guest facilitates a visit, and all the visit represents is fn(T) -> U

[wraith sequoia]

@thorn badger what does

E::value((self, visitor))
            .update_map((), |_, (this, visitor)| {
                RecoverableScope::<'ctx, E>::new_walk::<'_, '_, '_, '_>(this, visitor.cast()).cast()
            })
            .map((), |_, ((this, _), _)| match this.error {
                Some(err) => Err(StructWalkError { kind: err }),
                None => Ok(()),
            })
            .cast()

mean in impl Walker<...> for StructWalker<...>

[thorn badger]

It uses the struct walkers' recoverable protocol walk implementation to do a normal walk

The difference being a normal walk consumes the walker to perform the walk, a recoverable one only takes a mutable borrow and each call to new_walk is supposed to reset the walking state

[wraith sequoia]

Okay

So does a StructWalker walk the entire struct at once, or does it do like a deference thing to walk each element by themselves?

[thorn badger]

It's walks the entire struct in one go by starting a sequence scope which hands control to the visitor to pull items out like an iterator

It inverts control temporarily because the visitor likely has state it wants to track while pulling items out (like say a Vec of items)

It's not required to do though

I do quite a bit of control flow inversions in treaty to make the side with more info easier to write

It also follows serde's pattern which makes the adapter much easier

[modern jetty]

How's stuff going with treaty anyway?

[thorn badger]

I am currently side tracked making proc macros

[wraith sequoia]

any macros that implement StructTypeInfo, for reference?

[thorn badger]

The Walk macro in lib.rs

[wraith sequoia]

So SequenceScope is just Cajun Frog Style SeqAccess

Right*?

[thorn badger]

Yeah

[wraith sequoia]

Okay. So what like . Consumes It. And how

[thorn badger]

It gets passed when a sequence is visited

[wraith sequoia]

So do walkers implement it out of necessity?

[thorn badger]

The walker itself doesnt have to implement it, but yeah usually it does

And it only needs to if it's going to visit sequences

[wraith sequoia]

This confuses me so bad. It confuses me in the case of serde too, though. Cuz in my interpretation, something like a SeqAccess would just be for accessing concrete data structures in a way adapted to the API (literally giving access in a way the API understands, i.e. giving access to push so the API can add data in a way it understands). So in my head I assume it's implemented for the actual types themselves, like Vec or HashMap

But in reality it's more like changing the behavior of deserialization

[thorn badger]

Its a fancy iterator

The iterator with the extra state implements Iterator (analog is SequenceScope)

[wraith sequoia]

So ArrayWalker is like the IntoIterator of sequences

[thorn badger]

More or less

Though you just equated a struct to a trait

[wraith sequoia]

Well Walk is too vague as a noun but yeah

[thorn badger]

I took the term from tree traversal

[wraith sequoia]

Is there an implementation of Walk for Vec?