#Idea for better serialization framework

3760 messages · Page 4 of 4 (latest)

thorn badger
#

Currently no, but there would be

#

People usually say you "walk a tree"

#

The idea is as the walker walks, it visits things

wraith sequoia
thorn badger
wraith sequoia
#

Interesting.. ........

thorn badger
#

Walkers and builders can reuse each other

#

Which is nice

modern bone
#

What optimizations could you gain from using a ref to the vec instead of a slice?

thorn badger
#

Exactly what gets passed for each type I'm still not settled on

scenic swallow
#

Will this new serialization framework be zero-copy? No just curious 😅

long sigil
#

I believe frog has said as much

#

Serde is somewhat zero copy where possible

#

I do zero copy json quote often

thorn badger
scenic swallow
# thorn badger Yes, though probably not as much a rkyv

Ok I see, well if we could have such support it would be great I feel like I really have a use case for myself like I want to use a zero-copy deserializer for my app which also has a great support with actix. If you could provide such support it would be great to have. 😅

wraith sequoia
#

what's the point of seeds when it comes to builders

visual tartan
#

Is treaty materialized now

thorn badger
wraith sequoia
#

im having some trouble with my sequence definition...

pub trait SeqScope<'ctx> {
    type Error;

    fn visit_next<V>(&mut self, vis: V) -> Result<Option<V::Value>, Self::Error>
    where
        V: Visit<'ctx, ???, Self::Error>;

    fn size_hint(&self) -> Option<usize> {
        None
    }
}

how does your SequenceScope construct values while just returning Flows n stuff

thorn badger
thorn badger
#

In serde's case the Visitor doesn't have a generic

wraith sequoia
#

Ah is this one of those cases I should ask a question about the Builders

#

Atleast I knew ahead of time that my visit was kind of an abominable cross between the two

wraith sequoia
wraith sequoia
thorn badger
thorn badger
thorn badger
#

A sequence scope encodes the idea/ability of walking (iterating) a sequence of values that may be different types

#

In some sense it's a Iterator<Item = dyn Any>

#

Visitor is taken from the usage of serde where something consumes data from a deserializer

#

Written using the From/Into traits it's: From<dyn Walker> for dyn Visitor

wraith sequoia
#

So let me see if I understand... it's not the same as serde, where the return value lands at the callsite, but rather in the pocket of the visitor, so to speak

#

So Builder can save the visited value, and return it?

#

as a visitor

thorn badger
wraith sequoia
#

So i guess I understand that part, I still don't think I understand what it all would expand to. Is this like how you could use it at all?

struct Foo {
    id: i32,
    name: String,
}

let value = Foo {
    id: 420,
    name: "weed".to_string(),
}
let other = value.into_walker().walk(/* idk how the visitor would look but yeah */)
#

Like how would you obtain other, as a modified version of value, if the visitor modified it in some way

thorn badger
#

it acts as an out param

#

do to the need for object safety

thorn badger
tired warren
#

Is it already usable?

thorn badger
#

I'm currently like 2 layers of crates deep making things treaty needs as dependencies

dawn herald
#

lol

open oriole
last furnace
pearl zephyr
silver bolt
thorn badger
sharp condor
whole berry
#

any updates?

#

how many subcrates are there now

thorn badger
#

So many

modern bone
#

What started out as a replacement for serde has become a replacement for all of crates.io

whole berry
wraith sequoia
scenic swallow
# thorn badger So many

Sorry, but I am still wondering where is the project located and what is it called. 😅 So could you please provide a link to your project?

thorn badger
#

To be completely honest the repo itself is very not helpful at this point

#

It's the least documented of them all

#

So as it stands, treaty is waiting on supply to replace it's lifetime any system. I am also still working on effectful but that one is developed with treaty. I also have parsing and proc macro crates in the works for treaty's proc macros

#

And supply is being split into 3 crates: type_tag, supply, and lt_any

dawn herald
#

Frog on their way to replace half the Rust ecosystem:

sharp condor
wary tartan
whole berry
long sigil
#

i need to check whether io is still controlled by uk or not

#

we recently gave up our claim on the indian ocean island territory

scenic swallow
#

Sorry but I wonder folks is this thread not getting quite offtopic? 😅

long sigil
#

that's what threads are for :3

wary tartan
scenic swallow
# long sigil that's what threads are for :3

Yes, right but I just had this feeling from seeing all this different topics discussions here as well as seeing that we already have a #offtopic channel for offtopic discussions. So It kind of made me feel like that.

But anyways you folks better know than me (as I am not the mod) 😉 😅 .

modern bone
#

If you look hard enough, you can find other offtopic threads, hidden from the public eye. These usually don't get found by the spammers and trolls, so they stay nicer: that's intentional.

scenic swallow
# wary tartan I thought since there were already plenty of cat pictures, some discussion about...

Well Idk about the cat stuff 😅 but I kind of find this community really like safe place tbh like you feel really like at home. While on the other hand I have seen other communities like once I was part of this one webdev community because I also have the skill in web dev. And they were talking about weird stuff like one folk was talking about weird medicine they have to talk and it talks a long time to dissolve I mean why are we discussing these stuff here. Like the community felt really nostalgic to me and also I started having negative vibes just being there.

The only thing I found most here is anime Idk why maybe alot of anime fans here. Though I don't like anime myself (please don't hate me after this one). 😅

Though, I do fear one thing, like if this community grows it doesn't end up being like other communities though I don't mind change but not the negative/bad one.

scenic swallow
wanton basin
#

@long sigil, hey, this is a forum thread with a dedicated purpose. In the future, please avoid leaving that purpose behind.

#

This is actually just due to us not having more detailed moderation guidelines written out anywhere.

(Open issue, we know about it, but it may be a while before we can address it.)

wanton basin
#

This one is very much about @thorn badger's very WIP serialization framework

#

Generally speaking, we still expect people to not detract from the purpose of a thread they did not create.

#

I should probably start writing down moderation policy somewhere we can bikeshed. I guess I'll do it in a bit.

thorn badger
modern bone
#

Mod v. mod, who will win? (/j)

woven crow
reef edge
wanton basin
reef edge
sharp condor
woven crow
#

arena allocator

modern jetty
thorn badger
mellow thistle
#

life ferrisConcern

long sigil
#

Get rid of it

dawn herald
#

rip life

modern bone
#

If GOL is eating your resources and you can't stop it, try rebooting

modern jetty
#

how's this going then? :3

open oriole
#

:3

#

mrow

sharp condor
open oriole
#

:3

#

so wha the progress on this

thorn badger
#

I'm working on finishing the update to supply

#

Then I can use supply to replace some of the internals of treaty

dawn herald
#

:D

long sigil
#

Oh no

forest river
#

IMHO i think ssz is great!
?crate ssz

tribal rampartBOT
#
ssz

Simple serialization implementation

Version

0.2.0

Downloads

6 770

thorn badger
full hedge
#

Oh, sheesh. 367 days already

long sigil
#

Happy birthday

wraith sequoia
#

?crate supply

tribal rampartBOT
#
supply

Provider API for arbitrary number of lifetimes.

Version

0.2.0

Downloads

1 043

dawn herald
#

> 1K downloads

# 
impl<T, L> Reify<L> for T
where
    T: ?Sized + Tagged,
    L: LtList,
    LifetimesTagOf<T::Tag>: FirstN<L>,
    LifetimesTagOf<T::Tag>: FirstN<
        LifetimesOf<LifetimesTagOf<T::Tag>, L>,
        Lifetimes = LifetimesOf<LifetimesTagOf<T::Tag>, L>,
    >,
    T::Tag: WithLt<LifetimesOf<LifetimesTagOf<T::Tag>, L>>,
{
meager field
#

holy shit

#

HOLY SHIT

thorn badger
meager field
#

how did LLVM literally optimize everything out

thorn badger
#

its good

#

it also doesn't always work out

#

the addition of effectful did not help things

meager field
#

also i had a question about const_transmute

#

i thought any::type_id_of had some non-equivalence problems

#

like the type id of T not equaling type id of U didnt mean T != U in some cases

#

or something

thorn badger
#

and it is now more or less fixed

#

since to get a collision takes about 1 type per nano second for the entire length of the universe's lifetime

#

(its a u128)

meager field
#

oh good

#

so you wen't with a type erasure type of thing instead of const transmute?

#

also, how am i supposed to get the ../effectful dependency?

thorn badger
thorn badger
#

fair warning I haven't worked on treaty itself in some time so i don't remember if it was left in a working state

meager field
#

i'm not really looking to run it or use it but more to understand what generic programming that optimizes well looks like and works like

#

lol very readable test error

image.png
thorn badger
#

or really anything async

meager field
#

i dont even know what async does or works like

dense oasis
#

The short version is that fn() -> &'static T and for<'a> fn() -> &'a T used to be subtypes of each other. This, in principle, makes sense, since they're more or less two ways to write the same type.

However, they do not have the same typeid, so you end up with a funny situation where "the same type" has two different typeids.

#

I think we've since decided those two aren't mutual subtypes, so they're actually just different types and them having different typeids makes sense?

meager field
#

is that the only case or is there infinitely many more?

dense oasis
#

AFAIK infinitely many, but if I'm right we fixed them all

meager field
#

damn we done infinite work in finite time, that's a supertask

dense oasis
#

The cased would be the infinitely many different ways to write the same thing once you bring for<'a> into the picture

tired warren
#

oh no, all the crates are on gitlab, why oh why

#

setup github mirrors, maybe?

sacred shell
thorn badger
tired warren
#

nooooo

#

why not though, it should be trivial to write a hook to sync on commit/release

mellow thistle
#

what's wrong with gitlab?

tired warren
#

it being unconventional

thorn badger
#

I don't want to manage two different spots

mellow thistle
#

good ferrisSquee

tired warren
#

almost noone uses it, which makes interacting with it difficult, as I have all my env setup for github specifically

thorn badger
#

I like gitlab, if you don't, you don't need to interact with it

tired warren
# thorn badger I don't want to manage two different spots

I remember a guy who has gitlab being a mirror from github without ever needing to touch the former, so should be simple. I'll look into it at some point, will share if some simple script to just procedurally sync the state can exist

thorn badger
#

Plus I only use GitHub to report issues with other crates

thorn badger
mellow thistle
#

yeah github is not great

tired warren
#

offtopic, but what is even so good about gitlab, why use it over github?

thorn badger
#

I much prefer gitlab's UI and user management

#

Plus I use gitlab at work in addition to bitbucket

mellow thistle
#

plus it's not owned by Microsoft and aggressively pushing ai

thorn badger
#

Plus I have been using gitlab since 2017

thorn badger
mellow thistle
#

ikr

thorn badger
thorn badger
#

For the record I don't have an issue with using GitHub, I just choose to not use it for most of my projects

meager field
#

i just use it for the sake of my employer hopefully recognizing github so i get more credit in my future endeavors

#

i doubt they'll give 2 fucks tho

sacred shell
sharp condor
thorn badger
#

I think I'm going to give up slightly on async support in treaty, I'm going to just force all the futures to be boxed

#

That should make effectful tolerable to use

long sigil
#

I need to rework the tokenizer for sonny jim to be async

dawn herald
dawn herald
mortal python
#

trying to build docs for treaty rn and it doesn't work due to unresolved imports from effectful, is that something I can fix?

thorn badger
#

It may or may not actually build though

#

Considering you would need to find the exact commit treaty was using

#

Or I guess you could try the newest V2 version

#

With the new supply and ty-tag

#

Though that one only has the core primitives since I am redoing them

mortal python
#

I cloned treaty, effectful, supply and ty-tag, are there specific branches/tags I have to switch to in order to make it build?

thorn badger
#

well hard for me to say...

image.png
mortal python
#

I see :D

thorn badger
#

oh right I have uncommitted edits to supply, ty-tag, and effectful

#

welcome to my haphazard development of personal projects

#

treaty has never been in a state where I would even consider it alpha software

mortal python
#

rn I'm mostly interested in looking through it and understanding the idea/architecture, it seems quite interesting and I might want to implement something similar in other languages

thorn badger
#

I have had to explain the idea quite a few times

thorn badger
#

treaty has 2 basic protocols (parts of a schema) one is for a single value to be passed from a walker to a visitor (what I call the two parts), the other protocol is to allow a visitor to ask for multiple possibly different values from a walker in a sequence

#

negotiation happens via supply (in the new version) to allow asking a walker or visitor trait object "do you implement trait X"

#

a lot if treaty's in code complexity is not actually related to this but to my want to support async

mortal python
#

tysm for the explanation! I think I kinda understand the high level idea now, but I'll try to look at the code to get a better understanding of how it works in detail

thorn badger
mortal python
bronze dune
dawn herald
bronze dune
#

it was the only good control flow framework without any costs, rip

dawn herald
bronze dune
#

i wonder if we will ever see a production capable control flow framework in rust

dense oasis
#

Oh, nevermind, box pins

dawn herald
thorn badger
#

The blocking mode is zero cost

#

Which is the one I care about, async I want to have but I'm not concerned with it being blazingly efficient

#

If someone can figure out a way to do it without killing rustc and be more efficient than boxed futures then I am open to adding it. I made sure to leave effectful's API open to other possibilities

#

I would also be interested in doing some benchmarks at a later date to see how much overhead the boxed futures have over the blocking mode

thorn badger
thorn badger
modern jetty
#

seeing that C++ decides to box every coro, I don't think it will be a big deal

sacred shell
undone flax
#

coroutine probably

modern jetty
#

indeed

charred river
#

Please refer to them as corroutines from now

open oriole
#

:3

modern jetty
#

how's this going? @thorn badger

thorn badger
thorn badger
# 
use supply::prelude::*;

struct Demo<'s> {
    s: &'s str,
}

impl<'r, 's> Provider<'r> for Demo<'s> {
    // For the example we include both lifetimes even though only 's is used.
    type Lifetimes = l!['r, 's];

    fn provide(&'r self, want: &mut dyn Want<Self::Lifetimes>) {
        // The N<1> is to make it &'s str.
        want.provide::<ValueInferTag<N<1>>>(self.s);
    }
}

#[test]
fn example() {
    // A string that lives longer than demo.
    let s = String::from("hello");

    let s_ref = {
        let demo = Demo { s: &s };

        // The N<1> here changes the lifetime of the &str to
        // be &'s str instead of &'r str. That way we can return
        // the borrow from this scope.
        demo.request::<Value<&str, N<1>>>()
    };

    // This is a reference to s as expected.
    assert_eq!(s_ref, Some("hello"));
}

supply works again ferrisCat

# 
#[test]
fn example() {
    let mut demo = Demo { value: 0 };

    // Erase the type so the following doesn't know it's a Demo.
    let p: &mut dyn for<'r> ProviderDyn<'r> = &mut demo;

    // Access demo as a &dyn A to call get_num.
    assert_eq!(p.request::<Value<DynA>>().unwrap().get_num(), 0);

    // Access demo as a &mut dyn B to call set_num.
    p.request_mut::<Value<DynB>>().unwrap().set_num(5);

    // Access demo as a &dyn A to call get_num again.
    assert_eq!(p.request::<Value<DynA>>().unwrap().get_num(), 5);
}

trait A {
    fn get_num(&self) -> u8;
}

#[tag(remote)]
type DynA<'r> = &'r (dyn A + 'r);

trait B {
    fn set_num(&mut self, value: u8);
}

#[tag(remote)]
type DynB<'r> = &'r mut (dyn B + 'r);

dynamic traits go brrrr

dawn herald
#

Some crates do this and it really pisses me off every single time

#

I don't even know how they do it, but there are a few crates that have types which aren't accessible from their actual location, only the prelude

dawn herald
thorn badger
dawn herald
#

They just managed to make it re-export things that are available through no other path into the goddamn crate

thorn badger
#

I mean that I have checked that it's reexporting from the public spots

thorn badger
dawn herald
#

Groan

#

Really does not like typing prelude to access things :c

thorn badger
#

im not sure if im going to keep the tag macro being reexported because it needs you to do

#[tag(root = supply::ty_tag)]
// ...
dawn herald
#

Looks at repository

#

merged from wrong branch

thorn badger
#

yeah I was accidentally working on my 0.3.1 branch

#

rustdoc please

image.png
#

just let it link to the macro's page

meager field
#

does rust have traits as generic arguments?

#

like ```rs
trait Foo<T>: T {}

impl<T> Foo<From> for T {}```

undone flax
#

no

meager field
#

i just saw ```rs
#[tag(remote)]
type DynA<'r> = &'r (dyn A + 'r);

trait B {
fn set_num(&mut self, value: u8);
}

#[tag(remote)]
type DynB<'r> = &'r mut (dyn B + 'r);


and went "why dont you just" ```rs
#[tag(remote)]
type DynT<'r, T> = &'r (dyn T + 'r);``` and realized `A` and `B` werent generic types but rather traits
thorn badger
#

so much work to do ferrisMelt

thorn badger
#

I am once again asking myself if effectful is worth it

dawn herald
#

Maybe

meager field
#

what's it do

wraith sequoia
thorn badger
thorn badger
# wraith sequoia What's the current hell 
fn walk<'r>(
        mut self,
        mut visitor: DynVisitor<'r, L, E>,
    ) -> ThunkMaybeSend<'r, E, Result<Self::Output, Self::Error>>
    where
        Self: 'r,
    {
        Ctx::state((self.0.unwrap(), visitor))
            .into_thunk_maybe_send()
            .update_maybe_send(|ctx| {
                let this = &mut ctx.state.0;
                let mut visitor = &mut ctx.state.1;

                let x = visitor.request_mut::<meta::Value<ValueTag<ValueSrcSend<&mut T>, E>, (_0, _0, OffsetLen<_1, L::Len>)>>();
                let y = x.unwrap();
                
                let x = HasSend(&mut this.0);
                
                y.visit(x)
                    .map_maybe_send(|_| Ok(()))
            })
            .no_state_maybe_send()
}
#

The thunk stuff

wraith sequoia
#

lol are you also making it generic over send/syncness?

thorn badger
#

Since none of serde has Send bounds

wraith sequoia
#

yeah but damn

thorn badger
#

Im thinking I should just give up on async support

dawn herald
#

:c

#

RIP streaming

thorn badger
#

I do have one idea left for effectful to try

#

But my hopes aren't great

dawn herald
#

I was kind of really hoping to use treaty for async, streamed deserialization of a very large list

#

Like, "hundreds of megabytes" large

thorn badger
#

If we could come up with a way to make it not destroy the API

#

Like I could split all the protocols into an async version

#

Basically what every other crate does

#

Have one async API and one sync API

modern jetty
#

I have a feeling like it would be quicker to implement or even wait for someone else to implement effects in the compiler

#

in terms of development time

wraith sequoia
#

what are effects?

wraith sequoia
thorn badger
mellow thistle
thorn badger
#

Also fallibility is usually included

wraith sequoia
#

That's like the gist of it afaik, but how is it usually implemented?

#

Or "used" i guess

dawn herald
#

@thorn badger What problems have you been running into so far?

wraith sequoia
#

The way I've heard it it's basically like returning something and then collapsing it to a desired effect

thorn badger
#

Which currently I need unsafe at every impl site

#

And every type needs it basically

thorn badger
thorn badger
thorn badger
meager field
thorn badger
meager field
#

i mean it would be great to be generic over asyncness, constness, infallibility etc. etc. all at the same time to avoid code duplication

#

but at some point wont it get so complex its not even worth to implement stuff for all cases?

thorn badger
#

If that user doesn't support both even if they could then their stuff becomes useless for half the API

dawn herald
#

^ treaty itself is just glue

thorn badger
#

It's a problem I don't know if I want to deal with since it means likely that treaty's async will be kind of useless

wraith sequoia
thorn badger
#

They all do it in a different way

wraith sequoia
thorn badger
wraith sequoia
thorn badger
#

Then conrad's

wraith sequoia
#

This has to be insider behavior

#

I'm gonna look @ it tho

thorn badger
#

Or well

wraith sequoia
#

When the readme mentions maybe-impls, is it talking about that proposed ?const/?async syntax

thorn badger
wraith sequoia
#

Cuz I agree I think that shit is uggo as fuck

thorn badger
wraith sequoia
#

If I was in the KGI I would say each function should have a default coloring that can be changed/gleaned by compiler inference or through a specific keyword that enables that inference

full orbit
thorn badger
#

my new try with witness proofs seems to be going well

full orbit
full orbit
thorn badger
# full orbit could you explain what are you doing right now a little?

so blocking code (in particular anything interacting with serde) doesn't know if it's types implement Send, but async runners usually need Send for any kind of ergonomics. This is a problem effectful has had since the beginning.

The current try is to have types that act as witnesses that a given T implements Send without me needing to write T: Send. We can then compose these witnesses together and transfer them around the code via associated types and generics

# 
type Pause;

type PauseWitness: ProofIsSend<Self::Pause, Self::NeedSend>
where
    OutputWitness: ProofIsSend<Output, Self::NeedSend>,
    StateWitness: ProofIsSend<State, Self::NeedSend>;

This is a Pause associated type where we can prove (via a witness type) that it implements Send if we can be given a witness that Output and State implement Send

full orbit
thorn badger
# 
fn thing<'a, Env: Environment, T, W>(x: T) -> Thunk<'a, Env, i32, u8>
where
    T: core::fmt::Debug + 'a,
    W: ProofIsSend<T, Env::NeedSend>,
{
    Ctx::new(x, 0)
        .thunk()
        .with_witness::<W, _>()
        .update(|ctx| {
            *ctx.state += 1;

            dbg!(ctx.output);

            Ctx::output(42).thunk()
        })
        .update(|ctx| {
            dbg!(&ctx);

            Ctx::output(ctx.output + 1).thunk()
        })
}

code taking a generic needs a witness passed in with the form W: ProofIsSend<T, Env::NeedSend>,

#

the Env::NeedSend says if we need the proof or not

#

ProofIsSend<T, No> is trivial for all types

#

since we don't have to prove anything

#

a ProofIsSend<T, Yes> needs something to know that T: Send

# 
/// Wrapper that implements `Send` if the `Witness` proves `T` does.
#[repr(transparent)]
pub struct SendShim<T, Witness = SendWitness>(pub T, PhantomData<fn() -> Witness>);

// SAFETY: The witness proves that T: Send.
unsafe impl<T, Witness> Send for SendShim<T, Witness> where Witness: ProofIsSend<T, Yes> {}

we can then regain a true Send bound via a shim if something acts as a proof

# 
let send_capture = SendShim::new_with_witness::<CaptureWitness>(capture);

SendShim::new(Box::pin(async move {
    let ctx = FutureShim(pause).await;

    f(send_capture.into_inner(), ctx)
}))

like here is some async code that has a CaptureWitness that knows that capture implements Send (it has CaptureWitness: ProofIsSend<Capture, Yes>) it uses this to wrap it in the shim so the async block is send

# 
error[E0277]: `Rc<{integer}>` cannot be sent between threads safely
   --> src/lib.rs:515:35
    |
515 |         let x = thing::<Async, _, SendWitness>(Rc::new(43)).into_future().await;
    |                                   ^^^^^^^^^^^ `Rc<{integer}>` cannot be sent between threads safely
    |
    = help: the trait `Send` is not implemented for `Rc<{integer}>`
    = help: the trait `send_proof::ProofIsSend<T, F>` is implemented for `send_proof::SendWitness`

error[E0277]: the trait bound `(): send_proof::ProofIsSend<_, send_proof::Yes>` is not satisfied
   --> src/lib.rs:515:35
    |
515 |         let x = thing::<Async, _, ()>(Rc::new(43)).into_future().await;
    |                                   ^^ the trait `send_proof::ProofIsSend<_, send_proof::Yes>` is not implemented for `()`
    |
    = help: the trait `ProofIsSend<_, send_proof::Yes>` is not implemented for `()`
            but trait `ProofIsSend<_, send_proof::No>` is implemented for it

if you try to use an invalid proof you get an error like these

thorn badger
# 
        Ctx::state((SendShim::new_with_witness::<Witness>(self.0.unwrap()), visitor))
  .thunk()
  .with_witness::<SendWitness, SelfWitness>()
  .update(|ctx| {
      let this = &mut ctx.state.0;
      let visitor = &mut ctx.state.1;

      let x = visitor.request_mut::<meta::Value<ValueTag<&mut T, E>, (_0, _0, OffsetLen<_1, L::Len>)>>();
      let y = x.unwrap();
      
      y.visit(this)
          .with_witness::<VisitResultWitness<MutWitness<Witness>>, SelfWitness>()
          .map(|_| Ctx::output(Ok(())))
  })
  .no_state()
  .with_witness()

A real example from inside of treaty

modern jetty
#

I hope no one actually has to write code touching this outside treaty

thorn badger
#

since thats the only time code needs to be abstracted over it

modern jetty
#

okay, so this is pushed to the format ((De)Serializer) side, not the type side, got it

thorn badger
#

(since they count as a format)

full orbit
thorn badger
#

if we tossed out lifetime preservation and async, treaty would be much closer to serde in complexity

#

oh also I guess if I made everything use futures

#

but then I loose good codegen for blocking

full orbit
thorn badger
#

the single function being basically

let new_thing = transform(old_thing).unwrap();
#

since treaty doesn't distinguish between serialize and deserialize or format vs rust types

thorn badger
full orbit
thorn badger
#

its a anything -> anything mapping system

thorn badger
full orbit
#

man the more I hear you talk about treaty the more I want it 🥲

thorn badger
#

and of coarse no data model is given

#

you can pass literally any rust type between the two sides

#

and perform any logic

full orbit
#

so it do way more then what serde does which is (de)serializing one type to another...
it can actually be used to transforming normal types between each other as well...

thorn badger
# full orbit so it do way more then what serde does which is (de)serializing one type to anot... 
let mut my_str = String::from("hi");

// We make a waker that will emit the my_str to the visitor.
let walker = ValueWalker::<L0, MyTag, _>::new(my_str);

// We make a builder that will accept a String (tagged with MyTag);
let mut builder = ValueBuilder::<L0, MyTag, Blocking>::from_seed(())
    .into_ctx()
    .output;

// We make a visitor from the builder that the walker can emit into.
let visitor: DynVisitor<_, Blocking> = builder.as_visitor();

// We walk the input data. In this case this just gives ownership of my_str to the builder via the visitor instance.
let _ = walker
    .walk(visitor)
    .into_ctx()
    .output;

// We finish building the String (we check we got a value).
let z: String = builder.build().into_ctx().output.unwrap();

// Print the my_str that passed all the way through the transformation.
dbg!(z);

here is an example I use for testing (this is what the transform function does internally)

thorn badger
#

so like json -> toml

full orbit
thorn badger
#

so I have (the framework for) adapters for serde serializers and deserializers and types

#

without that there is 0 chance treaty could gain any market share

full orbit
thorn badger
#

I have a lot of hard goals for treaty

full orbit
#

I mean I doubt treaty will any time soon replace the serde even with the serde support... but this will ease the changing process easier

thorn badger
#

though the core design is actually still basically what my first comment of this thread proposed, its just all the details of how to actually get rustc to accept it

thorn badger
#

one thing I think I could make a killer feature for it is to have automatic emitting of struct docs so you can write a struct to toml and have docs with it

full orbit
#

yeah that would be a fun thing... but doesn't that feel a little out of place for treaty..? I think of treaty as the library doing the transformation between x and y... docgen seems a little strange here... maybe instead add a way to give extra info to a type that then we can use to do all this cool stuff with?

thorn badger
#

the toml stuff just has to accept the comments meta data and write them out

#

basically I can treaty rust code as itself an information format

full orbit
thorn badger
#

yep

#

treaty is really a reflection crate in disguise

full orbit
thorn badger
#

give me another 50 years (like fusion)

#

once I have an alpha put together development should be faster

full orbit
#

thank you for the work you are putting btw... it was a blast understanding what you trying to achieve

thorn badger
full orbit
meager field
thorn badger
undone flax
#

is it a proc macro or is it a declarative macro that you use yandros crates for derive syntax

meager field
# thorn badger I enjoy melting peoples brains with my cursed code

this type of "associated type" and "proof trait" programming is probably well-defined but in practice it occurs in so little contexts people dont get the chance to get used to them, if i had more occasions where i could implement stuff using those i probably could remotely understand what your code does xd

thorn badger
undone flax
#

lol ferrisClueless

#

when i first saw treaty the #[derive(Foo!)] stuff was wacky

#

now i forgor the yandros crate name lol

meager field
#

wow i literally mixed the channels together

#

great

red kettle
#

@thorn badger why can't you use RTN for this stuff

red kettle
glacial terrace
#

oh wow

#

that was sooner than I expected

#

how fun

long sigil
#

RTN cannot be used to bound the output futures from calling AsyncFn* traits yet
Aww

thorn badger
red kettle
#

@thorn badger it's literally supposed to help with the Send-bound problem?! I mean I only got so far into trying to figure out what the hell you're doing so ferrisCluelessest

thorn badger
#

For the same code

red kettle
#

oh noooooo effect generic

#

well

#

not really but

thorn badger
red kettle
#

great

#

so like

Rust now supports AFITs and RPITITs, but we currently lack the ability for users to declare bounds on the anonymous types returned by such functions at the usage site. This is often referred to as the Send bound problem, since it commonly affects futures which may only conditionally be Send depending on the executor that they are involved with.
is there no way you can turn this abstraction into an RPITIT or AFIT

#

and allow the sendness to happen at usage sites

thorn badger
#

And involved unnameable closure types

red kettle
#

oh cool

#

right.

#

okay.

thorn badger
#

And unnameable async blocks

red kettle
#

I now understand your problem.

#

somewhat.

#

....

#

...damn.

thorn badger
#

Don't worry it is now "solved" at least more solved than it was

#

The witnesses work really well

dawn herald
red kettle
#

@thorn badger hm, is ty-tag really necessary?

thorn badger
red kettle
#

squints

#

what does "lifetime-enabled" mean

thorn badger
red kettle
#

well

#

you mention the 'static-lifting alternative for TypeId?

thorn badger
#

Yes

red kettle
#

but then you mention that the problem with such erasure is that it only allows erasure if all lifetimes are a single 'a

#

but that doesn't seem instrumental

thorn badger
red kettle
#

right

thorn badger
#

Namely better any

thorn badger
red kettle
#

uhh

thorn badger
#

Is the question why I don't use a single lifetime type erasure?

red kettle
#

Let's go with

#[must_use]
#[inline(always)]
pub fn of<T>() -> TypeId
where
    T: ?Sized,
{
    trait NonStaticAny {
        fn get_type_id(&self) -> TypeId
        where
            Self: 'static;
    }

    impl<T: ?Sized> NonStaticAny for PhantomData<T> {
        #[inline(always)]
        fn get_type_id(&self) -> TypeId
        where
            Self: 'static,
        {
            TypeId::of::<T>()
        }
    }

    let phantom_data = PhantomData::<T>;
    NonStaticAny::get_type_id(unsafe {
        mem::transmute::<&dyn NonStaticAny, &(dyn NonStaticAny + 'static)>(&phantom_data)
    })

}

"why isn't this the entire thing?"

thorn badger
thorn badger
red kettle
#

AH!

#

RECOVERING THE LIFETIME

#

okay

#

cool

#

thanks.

thorn badger
red kettle
#

yes, that isn't std::any::TypeId::of

thorn badger
red kettle
#

yes.

#

are you saying we're talking about some hypothetical type which can never be lifted to 'staticever

#

not even via the lifetime transmutation?

thorn badger
#

Not that the idea is

red kettle
#

It is not.

#

It does what it is intended to do.

#

It does not happen to suffice for your desires, but not because it fails at its task, merely because it deliberately ignores something you try to encode.

thorn badger
#

wait the static bound on self adds an implied bound, ignore me

red kettle
#

I assure you this code compiles. :P

#

and functions

#

in its horrible gremlinesque way

thorn badger
#

so yes it works, its just unhelpful

#

since you can't use it to downcast

thorn badger
#

in the grand scheme of things its not that bad

red kettle
#

I am basically going through your API surface and identifying how many levels of stretch goal treaty is on by now

#

<_<

#

and supply is obviously core to the entire thing, and it depends on ty-tag

#

but effectful has me going "hmm"

thorn badger
red kettle
#

Okay cool.

thorn badger
#

it abstracts away the async-ness

#

while preserving perfect code gen for non async code

#

at the price of adding a whole monadic structure to the code

thorn badger
#

its just the ever expanding needs to support them

red kettle
#

then I suppose I'm reinterpreting the original proposal with a leaner set of goals in mind.

thorn badger
#

if we change the goals then indeed treaty doesn't need to look like it currently does

#

im going for the "best" a serialization framework could be to my knowledge

red kettle
#

trying to imagine if there's a way that effectful and the rest could be segmented out or if they're forever married in API surface

thorn badger
#

effectful is already it's own crate

red kettle
#

well right now treaty depends on it

thorn badger
#

if you mean so that treaty doesn't have them in it's public API no

red kettle
#

yeah.

#

that's what I was wondering

thorn badger
#

it can't happen without giving up something

#

if someone can devise a way it could be I would be amazed, since this is the result of almost 2 years of thinking about the problem

red kettle
#

I'll take a look and start asking around

#

you're no slouch by any means

#

but there are people who have been thinking about this problem for 5 years

thorn badger
red kettle
#

Well, supply being its own thing and treaty depending on it makes sense to me

#

since it's basically a reimplementation of a stdlib API

#

that was

thorn badger
red kettle
#

not included in stdlib later

#

for Reasons

thorn badger
#

std doesn't need it beyond that

red kettle
#

I mean yes

thorn badger
red kettle
#

I'm basically saying that the only reason it shouldn't be in std is the lang might need to evolve a bit further for it to approach its Final Form, API-wise

#

because like

#

...

#

sufficiently generically useful APIs should... exist in std...? if we know what they should look like?

#

I mean the entire point of having a lean core library is that it should still provide basic tools, right? ferrisSweat

#

surely T-libs-api won't

#

anyways

#

moves along

thorn badger
open oriole
#

get more eyes on it idk

#

to help

thorn badger
#

if someone else wants to sure, i don't want to use reddit

#

plus thats something I would only do once I got to an alpha

open oriole
thorn badger
thorn badger
#

witnesses fixed my issues

#

at least the ones I had

#

so now we are back to uncharted territory

#

still dont know if using effectful will make other people go crazy or not

#

but 🤷‍♂️

#

the price to pay for async ferrisPuppyEyes

# 
Ctx::state((self, walker))
    .thunk()
    .with_witness::<SelfWitness, SelfWitness>()
    .update(|ctx| {
        let (this, walker) = ctx.state;

        if let Some(walker) = walker.request_mut::<meta::Value<
            &mut dyn Hint<L, ValueTag<TagOf<&mut T::Tag>, E>>,
            (_0, _0, OffsetLen<_1, L::Len>),
        >>() {
            dbg!("got walker");
            walker.hint(DynVisitor::new(*this))
        } else {
            Ctx::output(VisitResult::Control(Flow::Done)).thunk()
        }
    })
    .no_state()

its not that bad ferrisClueless

# 
async {
  if let Some(walker) = walker.request_mut::<meta::Value<
      &mut dyn Hint<L, ValueTag<TagOf<&mut T::Tag>, E>>,
      (_0, _0, OffsetLen<_1, L::Len>),
  >>() {
      walker.hint(DynVisitor::new(self)).await
  } else {
      VisitResult::Control(Flow::Done)
  }
}

is the async version

open oriole
open oriole
#

im curious wha nightly features would make this easier/make this project easier

thorn badger
thorn badger
#

nightly vs stable hasn't ever been an issue on this project

thorn badger
open oriole
#

o oki :3

open oriole
thorn badger
#

along with treaty supporting arbitrary lifetimes and not distinguishing between serialize and deserialize

open oriole
dawn herald
#

treaty not distinguishing between de/serialize, means that you can use it to directly convert between formats without having to go through an in-memory intermediate value like you would have to with serde

wraith sequoia
#

Gotta do something about that dyn Hint though mate needs an alias

dawn herald
full orbit
#

One question, I find serde very dependent on recursion and as a result can use a lot of stack
This can lead to stack overflow when (de)serializing some big types... Will treaty perform better on these scenarios?

thorn badger
#

What I may be able to do is actually make an effectful environment that does it seamlessly

#

Since the async one is close other than all the poll functions nesting

full orbit
#

Have you checked miniserde to see how they are doing it without any recursion?

thorn badger
#

Meaning it needs alloc

full orbit
thorn badger
#

it ruins my perfect code gen

#

because of the allocations

full orbit
# thorn badger It would need to be opt in

Even if it's opt in it would be still so useful...
Sometimes we can't just optimize size of types that we (de) serialize...
For example when we are interacting with external APIs... A good example is telegram api, it's filled with useless deprecated stuff that they just don't get rid of (because they don't want to break old bots) serde have problem with it... Having this could help there...

dense oasis
#

There's nothing else that could obviously be used as a stack

#

No recursion and you pass in a block of memory to be used for stack space would be workable

next tundra
#

So true I understood what that meant

thorn badger
#

.

thick summit
#

how is it going

#

with treaty

thorn badger
#

This question should not be asked ferrisClueless

thick summit
#

Ok

thorn badger
#

I have been quite busy with work so haven't worked on it much

#

This weekend should be a good time to though

full orbit
thorn badger
modern jetty
#

treaty is a psyop to make developers sad whenever using serde even though it's good enough and the rough edges can be worked around

sharp condor
#

Im sure that the frog will bless us soon enough ferrisCat

wraith sequoia
dense oasis
#

Given that I'm reasonably sure serde's visitor could be replaced by an enum, yes, it probably can

wraith sequoia
dense oasis
#

I have some concerns around MapAccess and I never tried it in practice

wraith sequoia
dense oasis
#

Amusingly, the fear I had around MapAccess did not actually pan out

dawn herald
#

lol

thorn badger
thorn badger
#

I'm still trying to work out a good way to have both the recursive visitor and a custom stack visitor

wraith sequoia
#

I'm kind of a brainlet though so it's like a caveman trying to carve a stone wheel out of teeth

wheat prism
pearl zephyr
#

I can't wait for this facet competitor

next tundra
#

Fasterthanlime reference

full orbit
thorn badger
dawn herald
thorn badger
dawn herald
#

💀

long sigil
#

becomes crab

pearl zephyr
#

(For me, I can just lie down in bed tired, and never have to move my arms or anything. So relaxing)

full orbit
#

But seriously take it easy for some time... Hope you get better

sharp condor
thorn badger
sharp condor
long sigil
#

but then you likely couldn't go to the bathroom by yourself which is less fun

quaint raft
# 
frog.arm as _
#

now it has a cast ferrisCluelessest

silver bolt
quaint raft
#

is frog part of the "as" haters club? ferrisCat

open oriole
#

one day :3

#

Also wtf is a facet

#

cc: @thorn badger how does facet compare to treaty :p

pearl zephyr
#

Treaty better cause Frog

open oriole
wheat prism
undone flax
#

does facet support lifetimed downcast

south terrace
tired warren
#

is this dead or?

modern jetty
#

yes

pearl zephyr
#

@thorn badger Do you plan to continue this sometime, or is it forever unfinished?

thorn badger
pearl zephyr
#

That's understandable! There's a lot that goes into this, and the last thing you want/need is burnout

thorn badger
#

Like at the moment I have been focusing on the design of a code formatter

thorn badger
pearl zephyr
#

What I'm taking from this is, wanting to focus on other stuff / not currently having the drive, doesn't necessarily mean never, just you need a really good break (am I interpreting this correctly?)

pearl zephyr
thorn badger
#

I haven't even managed to sell people the current version of supply ferrisMelt

thorn badger
thorn badger
thorn badger
pearl zephyr
#

It's a little sad, cause I don't feel facet (no offense to Amos) would be as optimized as yours (I could be wrong, this is just my initial unprofessional opinion)

thorn badger
thorn badger
#

Which isn't something that can be really tested until everything is done

#

At a base level I kind of feel Rust isn't ready for such a crate

#

It's possible, but only by abusing and building so much on top that it stops being reasonable

#

I could toss out features like async support, but then it's not the library I want to make

thorn badger
#

Plus I have had multiple times where I set everything up over weeks to only hit a rustc bug that means rewriting everything again

#

I just don't want to fight the compiler right now

thorn badger
long sigil
#

At a base level I feel like I'm not ready for such a relationship with madfrog

grizzled bison
thorn badger
#

Can we get a rust2? Then I can make it ferrisHmm

#

We just need to keep an eye on which language will be the next big thing so I can make the serde equivalent for it

thorn badger
wraith sequoia
thorn badger
#

And my work code is code I actively like working on so it's not a case of "it's just random code I happen to write"

#

"A farmer hired a man to work for him. He told him his first task would be to paint the barn and said it should take him about three days to complete. But the hired man was finished in one day. The farmer set him to cutting wood, telling him it would require about 4 days. The hired man finished in a day and a half, to the farmer’s amazement. The next task was to sort out a large pile of potatoes. He was to arrange them into three piles: seed potatoes, food for the hogs, and potatoes that were good enough to sell. The farmer said it was a small job and shouldn’t take long at all. At the end of the day the farmer came back and found the hired man had barely started. “What’s the matter here?” the farmer asked. “I can work hard, but I can’t make decisions!” replied the hired man." - Someone

I feel like this quite often (one of my favorite)

thick summit
thorn badger
#

Motivation being I dislike rustfmt in the edge cases

thick summit
#

nested languages??????? that would be HUGE

#

for markdown especially

thorn badger
#

Yeah I want this formatter to be able to format doc comments and their code blocks all in one go

zealous briar
#

wow

#

awesome

thorn badger
#

As always I don't know if I can reasonably do this, but this is the task I have set out to think and research

thick summit
#

have you seen topiary? that's basically what it is, it uses tree-sitter to define queries to format syntax nodes, handles broken syntax because that's just how tree-sitter works, and defining formatters is extremely easy: no need to write a parser, just write queries

#

Though, it's totally a WIP

zealous briar
#

ew tree sitter

#

i really need to get my lib going

thick summit
#

I want to see entirety of tree-sitter ported to Rust

zealous briar
#

yes

thorn badger
zealous briar
#

im doingb that

#

and more

thick summit
#

Every tree-sitter grammar is a huge 100k lines of code C file. and you need the tree-sitter runtime which is also a whole lot

zealous briar
#

comes with ast errors and recovery

#

have you seen my demo?

thick summit
#

haven't

zealous briar
#

#ai-quarantine message

#

here e.g

thick summit
#

yay

thorn badger
#

Currently my main question to answer is how to represent a document as a data structure that makes it easy to work with (parse, rearrange, insert, remove) and for an optimizer to decide the final layout for

thick summit
thorn badger
thick summit
#

it's nice that you can do it in rust instead of C as well

thick summit
thorn badger
#

I had heard of it but didn't really know what it was

zealous briar
zealous briar
#

plus recovery is ass

thorn badger
#

The prettier paper was an interesting read, but I want more expressiveness than it's document model allows

zealous briar
#

and no ast

thick summit
#

it's easy if you don't get any errors

#

and then, it's hard to debug

#

Very hard

zealous briar
#

yea i am doing alot of analysis on the grammar

#

so the proc macro can error

#

e.g. if lef recursion

thorn badger
#

At this point I have a list of about 30 papers/resources to read

zealous briar
#

and bounded recursion gets untangled automatically

zealous briar
#

it took me so long to even understand how they work at all

thick summit
#

how do you know all of these data structures? leetcode?

zealous briar
#

i know them from making this thing

#

cst's are nowadays represented as red green trees

thick summit
#

Oh, hold up, from intuition: is "red green tree" basically just a tree but syntax errors are their own node?

zealous briar
#

see rowan/roselyn etc

thorn badger
#

I happen to work researching large scale optimization, so I know quite a few graph things and decision algorithms

thick summit
#

Every time you have something like an enum Expr there is a variant called Expr::ErrorGuaranteed which just means we failed to parse it for some reason

#

But it doesn't abort the whole program

thorn badger
zealous briar
#

this is a good read

#

if you are interested

thorn badger
#

It's like a doubly linked tree but having been separated into two trees

#

And yeah the rust analyzer docs is what really made it click

zealous briar
#

Of these, only GreenNodes store the actual data, the other two layers are (non-trivial) views into green tree. Red-green terminology comes from Roslyn (link) and gives the name to the rowan library. Green and syntax nodes are defined in rowan, ast is defined in rust-analyzer.

Syntax trees are a semi-transient data structure. In general, frontend does not keep syntax trees for all files in memory. Instead, it lowers syntax trees to more compact and rigid representation, which is not full-fidelity, but which can be mapped back to a syntax tree if so desired.

#

the excerpt

thorn badger
#

I think focusing on how the red tree is usually built on demand distracts

#

makes it sound like it's just a tree + something

thorn badger
zealous briar
#

ohhh

#

interesting

#

the metadata question you mentioned yesterday?

thorn badger
#

Yeah

zealous briar
#

seems like a decent fit

thorn badger
#

My green nodes conceptually have lots of possible annotations

#

And I see a need for the red nodes to have a similar thing but only while it exists

thick summit
dawn herald
thorn badger
#

I'm sorry ryos ferrisNOOOO

tired warren
sturdy swan
#

Dependent types are nice aren't they

#

Until somebody abstracts everything to oblivion

open oriole
#

@thorn badger how is the formatter going?

#

Also I haven't even looked into the difference between facet and serde tbh what the diff

thorn badger
wheat prism
dawn herald
open oriole