Idea for better serialization framework

[visual tartan]

Do you srsly mean a parser which parses rust language?

[fair acorn]

yep

[modern bone]

No dependencies allowed, so she just copy-pasted their source in

[restive ravine]

more like "dawn was memeing about making a rust parser in a single file"

and wrote most of a lexer for it for the hell of it

[fair acorn]

does it work though

single file lexer would still be pretty cool

[restive ravine]

rustc_lexer is trivially single-filed tbh

[fair acorn]

we can force someone else to finish it /s

[restive ravine]

it's extremely simple

[last furnace]

How big would rustc in a single file be

[modern jetty]

cargo llvm-lines /hj

[silver bolt]

its not that bad

rust parsing doesnt seem too bad

i should attempt some time

[fair acorn]

treaty when

[thorn badger]

every time you ask I take another day off from working on it /j

[fair acorn]

take a day off work to work on treaty?

[visual tartan]

Lmao

I mean, it seems reasonably progressed (I only looked superficially tho)

[thorn badger]

https://gitlab.com/konnorandrews/treaty

[last furnace]

Step 1 of world domination is now complete

[fair acorn]

tests/* still use uniserde namespace btw

[minor cobalt]

MR to make the implementer! macro hygienic: https://gitlab.com/konnorandrews/treaty/-/merge_requests/1

[thorn badger]

thanks I did see it, I am currently rebuilding that macro

I did go ahead and merge it

I will make sure to keep it when I manually merge it with my working branch

[wintry path]

nice

[visual tartan]

Oh wow, you made it?

Awww test is yet to be updated

[modern bone]

And then you ruined it

[silver bolt]

guys its just a power of two number

[red kettle]

1000 more.

[fair acorn]

you dont get it

[untold crane]

aw tests are super broken rn cant test

lets see... sequences have regressed?

yep, and when they worked it's was super WIP. Looks like this needs some more time in the oven ^^

[thorn badger]

Yep, I rebuild how the trait up casting was represented

[visual tartan]

Please let me know when you update the tests!

[open oriole]

Any updates on this? @thorn badger

[thorn badger]

Last progress I made was to make sure async works with the API

I had some issues with the code gen because of the async support, but I think that's all fixed now

[open oriole]

Nice

When will it be ready in a alpha state ig

[visual tartan]

.

Why is this in help forum

[modern bone]

It was a discussion/code review originally

[visual tartan]

Ahh

[last furnace]

because it'll help us all

[thorn badger]

I had many productive suggestions

[visual tartan]

Ahhh

[thorn badger]

This was actually sort of a continuation of a thread that was made during the serde binary incident

[last furnace]

what proportion of those involve using dungeon cell unnecessarily

[thorn badger]

I only use dungeon-cell style cursed code for 1 thing in treaty

[visual tartan]

Gotta use more then, since it is going to need dark magic

[thorn badger]

I needed double sized fat pointers 😅

[last furnace]

why

[thorn badger]

I needed to type erase trait objects

[last furnace]

why

wait

what the fuck?

[visual tartan]

Wait wha

[thorn badger]

For the protocol lookup system

Basically runtime trait lookup

[visual tartan]

Does type erasing helps?

[thorn badger]

But in a way that llvm makes not runtime

[last furnace]

are you recreating a better Any

[visual tartan]

Or do you mean just passing around type information

[last furnace]

ogga booga?

[visual tartan]

O

[thorn badger]

One of my core ideas with treaty is when not using async (because it has issues out of scope) the serialization process is optimized almost perfectly

[visual tartan]

Runtime trait lookup but.. not runtime? Sounds like dark magic

[thorn badger]

As if treaty wasn't there

[visual tartan]

Ah, like optimization in specific case

[thorn badger]

Here let me get the inspiration for this system

https://docs.rs/gdbstub/latest/gdbstub/target/ext/index.html I found this crate while working on another project

[south terrace]

sometimes I think I know rust, and then I look at crates like treaty and reliaze I am only at the starting of a rabbit hole.

[thorn badger]

Treaty adds one layer to this idea

https://gitlab.com/konnorandrews/treaty/-/blob/main/src/protocol.rs?ref_type=heads#L1 is my minimal writeup on the technique

Which happens to be out of date in terms of the code it references

But the idea hasn't changed

[last furnace]

don't we all

[last furnace]

have you taken your DIDETs today

[thorn badger]

Also yes treaty includes a Any that has multiple extra features like support for lifetime containing types

Which I use yandros' bijective traits to implement

[last furnace]

what what

bijective traits

[thorn badger]

https://gitlab.com/konnorandrews/treaty/-/blob/main/src/any.rs?ref_type=heads#L41

Which allows proving uniqueness

Without unsafe

Where as in dungeon-cell the same construct needs unsafe

[last furnace]

I'm so confused

this is beautiful

[modern bone]

I think the issue with treaty is that you (and maybe yandros) will be the only one capable of maintaining it, because it will be so arcane

[thorn badger]

Yeah...

[last furnace]

is that not an issue for serde at all

[modern bone]

It's not written in Black Speech, so not as much

[thorn badger]

You know you peeps interrupted me in the middle of trying to solve for the non linear vector field of the look_at function in 3d graphics

[last furnace]

isn't that your fault for being on discord

[thorn badger]

I cannot be held accountable

[last furnace]

what makes you think we can

I for one am excited for your villian arc where in five years where treaty rules the rust ecosystem, to see you silently sneak a wasm intepreter into the crate

[visual tartan]

Wh, you doing 3d graphics

[thorn badger]

It's been stuck in my mind from work

[visual tartan]

Hmm, wait Nonlinear vector field

what the

[thorn badger]

Technically I want the tensor

[last furnace]

I always forget people aren't in my time zone

[thorn badger]

Basically how the reference frame of a camera on a gimbal changes as it moves

[last furnace]

technically I am at work

[visual tartan]

Tensor is still linear tho

Maybe you are dealing with some distorted space time?

[thorn badger]

Wait a second I have solved this before... it's the local tangent plane coordinates but with a different orientation

[visual tartan]

Ahh

[dawn herald]

it turns out LLVM (the optimizer) is able to see through this style of runtime reflection. As such, we still gain the benefits of IDETs but with more flexibility.

[full hedge]

My algebra not mathing

@simple surge

Why is there a dude called silent* lmao

<@silent*>

[silver bolt]

to make this happen

[open oriole]

so real

:3

[visual tartan]

Argh, I was excited thinking there was some announcement by frog

[modern jetty]

how's this going then @thorn badger

[whole berry]

@thorn badger status update...?

last commit 2 weeks ago

[thorn badger]

Trying to decide if I should require every value to be Send

It's that or not supporting async

[modern bone]

When would Send restrict sync code?

(you never did get back to me when you asked earlier )

[thorn badger]

For example if you have a Rc

[modern bone]

Are you deserializing Rc? Ig it would make sense in some cases, actually

[thorn badger]

Yeah the restriction would be for any type touching treaty's API

[modern jetty]

I might have asked this before, but what do you mean by "supporting async"

asynchronous deserialisation?

[thorn badger]

Allowing things like async readers for parsers

Or async writers

[modern bone]

I suppose that'd be neat, but how often would it actually be useful? The same argument could be made for Rc, I suppose, but async reader/writers seems like a niche optimization, not a required feature

[thorn badger]

Think reqwest

And how it has to load the entire Json blob before deserialization

[modern jetty]

What's stopping async parsers requiring the values to be Send... or is this arcane magic?

[thorn badger]

The thing is to make async ergonomic I need to bound everything by Send

[modern bone]

This entire thing is arcane magic

[thorn badger]

So it's either have the bound or not have async

I tried really hard to have both (no bound and async)

[modern jetty]

I'd say have the bound, but I do a load of tokio programming so I've never done async without a Send bound (and don't think it's a good idea anyway)

[modern bone]

I suppose this could also be an issue regarding tokio interop, but personally I've never seen a !Send value get deserialized so idk how common that is

[thorn badger]

Yeah that's my thought, is serializable/deserializable things should be sendable because you know they are being serialized

My worry is this will become a fundamental limitation of treaty

[modern jetty]

this is already a fundamental limitation of a lot of libraries, it's nothing special and easily explained

[thorn badger]

But treaty

[modern bone]

IIRC the third option was having both sides be unergonomic?

(not advocating that, just curious)

[thorn badger]

Yeah

[modern bone]

So it's either sync is unrestricted, async is possible, or neither are clean?

[thorn badger]

Yeah

Oh also by supporting async I loose perfect code generation

But those cases were kind of rare in the first place so 🤷‍♂️

[modern bone]

Ooh, tipped the scales for me there, but I'm biased because I don't really use async anyways

[thorn badger]

Sadly llvm isn't great at optimizing away empty (as in no await) async blocks

[modern jetty]

serialisation/deserialisation is mostly done over IO boundaries, which should be asynchronous, so prioritising async (with Send, as that's what tokio needs) sounds sensible

[thorn badger]

Yeah that's what I'm leaning towards

[modern bone]

But most ser/de can't be done piecemeal, it has to be done linearly

[thorn badger]

the other reason I haven't worked on it recently was at work it was crunch week to get deliverables ready

[modern jetty]

Why?

[modern bone]

I suppose you could detect token boundaries and spawn a task to deser the token itself, but in any case you need context

E.g. 123 could be an integer, a part of a string, part of an ident, part of a float

[modern jetty]

Ah, I understand what you are saying, but this is for data being fed into the parser right? This just sounds like a way to do a bit of deserialisation while waiting for the whole response and therefore reducing latency.

[modern bone]

Mmm, that's valid, though I can't see that being very noticable in the grand scheme of things

Especially if you start parsing a token, only to realize you don't have all the info yet and have to backtrack

[modern jetty]

This conversation isn't going to be very productive anymore because I have absolutely no clue what the API looks like and this is getting into specifics

[modern bone]

Fair

[thorn badger]

Most parsers don't need backtracking

[thorn badger]

I will just do it to spite your feelings /s

[dawn herald]

If only Tokio didn't make it impossible to be agnostic over async runtimes

[woven roost]

Interesting stuff! Would love to read a small write-up to get the big ideas across.

[glacial terrace]

Me when I spread misinformation on the Internet

[modern bone]

Well, it's somewhat true

When crates rely on tokio::fs et. al., they are no longer independent of runtime

[fair acorn]

thats why we should all use io_uring instead of tokio::fs... oh wait now my code is not os agnostic

and also it looks like the main uring lib is tokio-uring so if you use that then have fun lmao

[glacial terrace]

That doesn’t make it “somewhat” true, since something can’t be “somewhat” impossible. you can make a Runtime trait like everyone does

[dense oasis]

Ye, the practical difficulties with defining a single universal Runtime trait don't really stop libraries from defining their own

[visual tartan]

Is async claiming another victim

[dawn herald]

Yes, it is

[last furnace]

Bites another one the dust

[visual tartan]

Low level asynchronous handling sounds like such a churn

[full hedge]

Wait until he tries to maintain both async and sync implementation

(I'm unexperienced. Don't mind me)

[thorn badger]

My week (really 2 weeks) of work is almost done

[open oriole]

:3

[thorn badger]

So I can finally work on treaty more

[open oriole]

It optimizations seem solid tbh

musli ones :3

[thorn badger]

One thing I find very interesting is how every serialization crate I have seen so far has two independent directions (encode vs decode, serialize vs deserialize)

Like why can't I just use a csv decoder directly into a Json encoder

I should add modes to treaty

[open oriole]

It uses some unsafe

but seemingly in a sane way

for optimization purposes :3

[thorn badger]

Don't look inside treaty

The double fat pointers are ...

[open oriole]

treaty is the name of the framework right

:3

[thorn badger]

Yeah

Unless someone steals it from me on crates.io

[open oriole]

wha the gitlab repo again

also first release when ;3

[thorn badger]

https://gitlab.com/konnorandrews/treaty

[thorn badger]

We are getting close I think I have worked out all the technical issues

[open oriole]

nice :3

[thorn badger]

https://gitlab.com/konnorandrews/treaty/-/blob/main/src/any.rs?ref_type=heads#L458

[open oriole]

[thorn badger]

Right now I am just working out how I want the included protocols to be used to describe things like structs and enums

Like how a struct is the protocol sequence:

• Empty tag of kind "Struct"
• Tag of kind "Type Name": name string
• Tag of kind "Type Id": TypeId
• Tag of kind "Field Names": sequence of field names ...
• sequence of fields

[dawn herald]

Yes

Ah fuck I was scrolled up lmfao

[modern bone]

cargo new treaty && cd treaty && cargo publish

[modern jetty]

you should def reserve that name @thorn badger

[last furnace]

yes

it's too important to be lost

[silver bolt]

that wont work

[modern bone]

Yeah yeah yeah, need a license and whatnot

[spring folio]

treaty is a cool name.. I'm gonna steal it

[south terrace]

already stole it

||/j||

[thorn badger]

there you go https://crates.io/crates/treaty

I did the thing

[mellow thistle]

omg

[last furnace]

Banned for squatting

[modern jetty]

omg treaty full release /j

[full hedge]

What was the difference between GitLab and GitHub that you choose the first?

[thorn badger]

I used gitlab for a long time in school for both classes and robotic clubs

And I prefer it's UI

Also gitlab actually got the name of merge requests right

[thorn badger]

This was before GitHub offered free private repos

I had one class use bitbucket because of that

[full hedge]

oh

how long ago was that?

[thorn badger]

2017 ish

https://github.blog/2019-01-07-new-year-new-github/ 2019 was apparently when they actually provided them

Also gitlab's organization management is (or at least was) a million times better than GitHub's

Made my life as the software lead on a robotic club way nicer

https://gitlab.com/usu-amr was the group for reference

Also I like that gitlab offers their software for free for self hosting

And that they manage the development of the free components using gitlab itself

[full hedge]

oh, I got interested in this

Is that aspect in GitLab and GitHub comparable, or it falls in the category of "Depends on what you personal taste is like"

[thorn badger]

I would say if you have a formal group of people Gitlab totally wins

[minor cobalt]

^

[open oriole]

@thorn badger Does Treaty steal borrow the optimizations that musli does?

https://github.com/cloudwego/sonic-rs

oo this is cool

:3

[thorn badger]

I haven't spent any time optimizing treaty other than it's unique trait lookup system

[dawn herald]

3. Get specific fields from a JSON with the blazing performance.
4. Use JSON as a lazy array or object iterator with the blazing performance.

"with the blazing performance."

[silver bolt]

the optimizations in this are not framework diagnostic

also, their benchmarks dont compare with rapidjson..... very sus

[open oriole]

Could probably ask the musli maintainer to do a benchmark on that on the issue page tbf

[thorn badger]

I stole borrowed the mode system

[modern bone]

When will you return it

[modern jetty]

keeping the thread alive lol

[digital gazelle]

⭐

[quaint raft]

the borrow has 'static lifetime

[high depot]

drop(Box::from_raw(&self)) its over

jsiouh wu uh083uh@@@@@

[sharp condor]

Not safe tho

[rustic glade]

all the code he writes is automatically wrapped in an unsafe block

[silver bolt]

i have a fork of rustc that forgets to check for safety of functions

[spring folio]

so it's crustc?

[silver bolt]

no thats mrustc

that turns off the borrowck

[modern bone]

mrustc isn't a fork of rustc, it's its own thing

[silver bolt]

no im saying that crustc == mrustc

mrustc still checks safety of functions

[modern bone]

But... they said your fork was crustc, you said no it's mrustc... I'm very confused

[silver bolt]

no they were saying its "named" crustc

i was saying that the real crustc is called mrustc

[modern bone]

But... what they were calling crustc was a fork with no safety checks, while mrustc is not that

[silver bolt]

yes but like crustc implies that there isnt a borrowck

[modern bone]

But that's not what they said

(I love arguments about frivolous things )

[silver bolt]

._ .

i just feel like crustc doesnt have the feel of a unsafe less safety check thing

[thorn badger]

Bump

[visual tartan]

Oh it alive

[open oriole]

Mrow

[open oriole]

How the status

[thorn badger]

I'm to the stage that I'm just working on some macro rules for structs and enums to demo the basic features

After that should be good for a alpha

[wanton basin]

[open oriole]

:3

[visual tartan]

How hard has this been?

[last furnace]

Poke

[fair acorn]

prod

[wraith sequoia]

So reading it over, there's a lot of catering to dyn. I'm not a dynamic-dispatch wizard yet, and I see it in a lot of larger libraries. Are the performance gains really that significant, or is it another reason, like ease of use for the client?

[thorn badger]

It won't have any dependencies

[thorn badger]

Also the trait objects optimize out

So there is no dynamic dispatch

[last furnace]

what's the cool link that explains this with a cool name

[silver bolt]

it has performance penaltys not gains

[thorn badger]

https://docs.rs/gdbstub/latest/gdbstub/target/ext/index.html

[silver bolt]

it has binary size gains

and here the compiler is all knowing

[thorn badger]

I'm currently fighting async block generation

If that gets solved then treaty can optimize to perfect code

Currently it optimizes to one stack frame that is like 300 bytes long

[dawn herald]

Oof

[wraith sequoia]

How do you go about defeating it

[thorn badger]

Unholy cursed code

[wraith sequoia]

I need to start writing code like that.

Too bad I don't know what the fucking significance of a HRTB is

[thorn badger]

#rust-discussions-1 message

[wraith sequoia]

You better tell me what those runes mean right now or you're getting burned at the stake for witchcraft

[wraith sequoia]

So what's the relationship between Builders, Walkers, Visitors and Effects?

[thorn badger]

A Builder constructs a value by having data injected via it's visitor, A visitor allows injecting arbitrarily shaped data, A walker walks over a data structure and for each thing visits it (via a visitor). And Effects allow being generic over async and blocking

So data flow is Walker -> Visitor

And almost all visitors are actually just builders

In serde land: Walker = Serialize/Deserializer, Builder = Serializer/Deserialize

In iterator land my_iter.for_each(|item| ...) the my_iter is the walker and the closure is the visitor

[wraith sequoia]

And so how do you plan for users to implement their own constructs (visitors, walkers)?

I see in some of your code visitors are functions that accept a DynVisitor<'ctx>

But I may have them misconstrued

[thorn badger]

It's going to have proc macros like serde as walkers and visitors combine nicely

[thorn badger]

How they get passed around is a little... complicated because of how it optimizes things

[thorn badger]

You might have noticed that the DynWalker and DynVisitor hold the same type

[wraith sequoia]

I did notice that it fucked me up. I was like why are there two definitions of the same type

[thorn badger]

yeah 😅 the AnyTrait type erases all notion of what the real value is

it even erases what traits the value implements

[thorn badger]

in previous versions i just passed around the &mut dyn AnyTrait directly and it made things way to confusing

walker: &mut dyn AnyTrait, visitor: &mut dyn AnyTrait

if you accidentally use one where the other is expected then weird things happen at runtime

[wraith sequoia]

Yeah, I'm sure. So the intention is that Builders construct the final value, but most of the time Visitors can also be Builders?

[thorn badger]

other way, builders can be viewed through the "lens" of a visitor

from an idea perspective builders add the build() function to the idea of a visitor

[dawn herald]

Finally got around to bookmarking the current URL for this project's repo lmao

[silver bolt]

what is a ! derive

[visual tartan]

Hmm I thought you unified builders and walkers as well

[wraith sequoia]

https://docs.rs/macro_rules_attribute/latest/macro_rules_attribute/

[silver bolt]

i see

[thorn badger]

No

[thorn badger]

They do use the same AnyTrait API to function though

[next tundra]

Meow

[wraith sequoia]

Where is AnyTrait defined?

[mellow thistle]

https://gitlab.com/konnorandrews/treaty/-/blob/main/src/any.rs?ref_type=heads#L146

[wraith sequoia]

I know it has something to do with higher_ranked_type!, but I don't think I fully understand the nature of TypeName and its members. What are they facilitating?

[thorn badger]

They allow giving a TypeId to non-'static types

The higher_ranked_type macro gives a bijection between a 'static type and a non-'static type mod the lifetimes it contains

Which allows for the safe transmutation (downcasting) based on the TypeId of the 'static associated type

[wraith sequoia]

Would it be asking too much for a breakdown of how it goes about that? I know I drop a lot of questions.

[thorn badger]

How it actually manages it is kind of technical and I don't have the time to write it down right now (currently at work) I can go get the conversation I had with yandros when he checked it

[thorn badger]

#dark-arts message (see this + the next message of mine + the thread off that message)

[wraith sequoia]

Thank you sm :3

[thorn badger]

Also the idea is based on the https://docs.rs/better_any/latest/better_any/ crate

Just formalized differently

[thorn badger]

I like that you ask questions. I'm waiting for the question that makes me go "oh I didn't think about that"

[wraith sequoia]

Most of my questions are really just centered around what you've already wrote. I want to get a better understanding of this code not only to improve my Rust abilities, but because I want to really understand how to harness DIDETs for the future.

I've actually learned a few things reading so far. I think the thing I still wonder is what part of this will apply to Readers and Writers

[next tundra]

Wtf is a DIDET yall need to chill with the acronyms

RPITIT was enough

[thorn badger]

dynamic inlinable dyn extension traits

[next tundra]

The what what what what what

[thorn badger]

WWWWW

Or the W5 for short

[next tundra]

WxW

[thorn badger]

[next tundra]

w3w

Gives me “k8s is just k9” vibes

[last furnace]

How do you not know about this incredible niche bit of rust

[next tundra]

I wish to know but I just got acronym jokes instead of an explanation :(

[last furnace]

There’s a good link explaining it somewhere

[thorn badger]

https://docs.rs/gdbstub/latest/gdbstub/target/ext/index.html

[last furnace]

Basically you just use dyn stuff in such a way rustc can devirtusise it

[thorn badger]

The dynamic part at the first is my addition

Which removes the big trait of extensions

[last furnace]

On an unrelated note I’m still trying to hunt down and ban the author of dungeoncell from adding crates to crates.io

Does anyone know where I can find them

[next tundra]

What they do

[thorn badger]

[thorn badger]

Exist

[last furnace]

So many crimes against rustc

[thorn badger]

I only murdered it like ... 6-7 times I'm sure it's fine

[last furnace]

Someone once told me their name rhymes with dog

[next tundra]

That read like it’s an upcoming feature, but it doesn’t the way you say it. Is it live?

[thorn badger]

I do need to report that OOM at some point

[next tundra]

Interesting

[thorn badger]

It's been possible in stable since 1.0.0

[last furnace]

The most frog thing of all

Forgetting to tell anyone about an Out of Memory error

[thorn badger]

I was more worried about it crashing my Linux

[next tundra]

OooOoOOm

Do we have

-linux

-gnu

[thorn badger]

Every time I opened neovim the system went ... dead

[next tundra]

Damn

[next tundra]

From wot, rustc?

[thorn badger]

Yeah

[next tundra]

Get moar ram that you otherwise would have had to get with VSC*de

[thorn badger]

I mean it ate 32 GB in about 10 seconds

[next tundra]

What’s your cpu? (Also damn that’s a lot of data)

[thorn badger]

It's a 3700x

[last furnace]

BLAZINGLY fast

[thorn badger]

After the incident I added https://github.com/rfjakob/earlyoom to my Linux install

[next tundra]

Wtf how can I not find a max ram spec on that shit nvm, joke ruined

[next tundra]

Adds GitHub to Linux

-tag

:(

-guthib

[zealous frigate]

guthib

[dawn herald]

Blazingly fast memory consumption 🚀

[wraith sequoia]

Have you implemented this yet?

[south terrace]

why such a fancy name to something so simple?

[fair acorn]

looks interesting, how did you make it dynamic?

[fair acorn]

ah theres a small writeup here

unreadable on my phone though

[thorn badger]

It's what AnyTrait implements

[thorn badger]

treaty provides the AnyTrait trait while in some sense allows up casting itself to any other trait object

[fair acorn]

split it into a separate crate?

[thorn badger]

The issue is the form of it in treaty is very much focused around the kind of traits the protocols are. Namely that they have a context lifetime associated with them

[fair acorn]

ah rip

dont imagine theres a way to be generic over lifetimes while being useful for that?

[thorn badger]

Not really

You would need a lot more powerful type system

[wraith sequoia]

Oh, okay. I asked cuz u still have that blurb about Implementation and Implementer, unless those aren't meant to be taken literally😅

[thorn badger]

Yeah I haven't updated some of the docs

That's what AnyTrait was called before

[south terrace]

how do you actually build the vtable if you dont have the concrete type, though?

or do you only support certain traits

[thorn badger]

Also at this point I don't have any interest in splitting treaty up, maybe in the future if people actually find it's abstractions useful for other things it will happen

[fair acorn]

fair enough

[thorn badger]

Treaty is still very much in the experimental stage of pushing rust to the limit

[fair acorn]

btw hows async lowering going

[thorn badger]

I have a double fat pointer which allows type erasing trait objects

[thorn badger]

I currently have it optimizing to perfect codegen so now I just need to clean up some of the effects API to not be so ... horrific

[fair acorn]

nice!

[thorn badger]

The process is you ask the trait for a impl of a trait object given a type id for the trait object, then if the value implements the trait it returns a type erased trait object which then gets downcast into the real trait object. All of this seemingly runtime specific stuff actually gets seen through by llvm and optimizes out

It still surprises me that llvm is able to see through all of this and optimize everything away

[thorn badger]

The type enrolls into the system when it implements the AnyTrait trait, specifying which traits it's going to support

Which also does allow it to expose some traits based on runtime state but I don't currently use that

[dawn herald]

@thorn badger

[fair acorn]

rust coom

[wraith sequoia]

This crate has taught me if I want optimizations from LLVM I should use match everywhere

[thorn badger]

I still need to report this OOM

[dawn herald]

What in the unholy Hells even happened there

[thorn badger]

I'm not sure

[dawn herald]

How did you make rustc consume 32 GB of RAM in 10 seconds?

[thorn badger]

I did something bad with GATS

[thorn badger]

The interesting thing is it prints the same error like 3 times then just gets stuck

[whole berry]

Llvm is just built different ig

I wonder how cranelift would handle this

[last furnace]

It might fail to see all the way through

[whole berry]

I doubt it would but I'd like to know how far it gets

Since it's opts is just "simple" IR rewriting

From what I've read

[last furnace]

I’d definitely suspect simple ir rewriting would be terrible at devirtualising

Could be a fun thing to find out

[silver bolt]

pretty easily

[next tundra]

Oh also now I look at this, can’t this be done with an impl now we have rpitit and only use &dyn when multiple types are actually involved?

Just pop on an impl Trait for &dyn Trait if that’s not already a thing

[thorn badger]

The original IDETs yes, though it makes the whole system more complicated

[next tundra]

Wouldn't it be the same just uh, different return type? Or do you want the trait to be object safe?

[thorn badger]

In particular for treaty it needs the AnyTrait, which can't be done with concrete types

[next tundra]

I don't know what AnyTrait is or what the structure of treaty is

So uh yeah I have no idea

[thorn badger]

There is also the issue of lifetime containing GATs having a bad habit of being unusable in async blocks, which is a problem for treaty in particular

[next tundra]

Lifetime skill issue?

[whole berry]

progress update?

[fair acorn]

i ate frog

[whole berry]

effect system

[modern bone]

"tests compile" is always a good sign

[thorn badger]

I finally have the perfect code gen I wanted

[whole berry]

https://tenor.com/bToTC.gif

[visual tartan]

Introducing some system to handle async vs. sync sounds like pain

[thorn badger]

It is painful

[wraith sequoia]

How'd u end up doing it

[thorn badger]

I replaced all the async blocks with closures then use adapters to chain them together

Basically what async was before edition 2018

[dense oasis]

Does that mean you don't support holding references across awaits, or did you only do that internally?

[thorn badger]

I have a special adapter for that where the closure can return a effective that borrows the context, but the output of the effective cannot

[dense oasis]

So, exactly like how async blocks work, neat

[thorn badger]

Yeah internally the adapter is just a async block when using async effects

[dense oasis]

In fact, that's also how async closures work. The returned future can borrow the arguments, but its output can't

[thorn badger]

https://gitlab.com/konnorandrews/treaty/-/blob/main/src/transform.rs?ref_type=heads#L17 is how it looks to use

https://gitlab.com/konnorandrews/treaty/-/blob/main/src/effect/async.rs?ref_type=heads#L154 is the async internals

https://gitlab.com/konnorandrews/treaty/-/blob/main/src/effect/blocking.rs?ref_type=heads#L134 is the sync internals

The upside of this system is it doesn't cause the buildup of mutable borrows when using only synchronous code that happens when using async blocks everywhere

Because async blocks are their own types they can't be optimized away usually

[thorn badger]

Also for users that don't care for this code gen improvement, they can use any future by using the from_future function of the effect, and can convert an effect to a future with into_future otherwise my testing would be so much more painful

[fair acorn]

[sharp condor]

[whole berry]

what are plans for first party parsers?

I'd like to see a first party json parser at minimum since while JSON isn't great it's by far the most universal

[thorn badger]

for the alpha I probably wont have any first class formats

or well I may have a demo one for RSV (and maybe SML and/or WSV)

[whole berry]

RSV?

[thorn badger]

https://github.com/Stenway/RSV-Specification

[whole berry]

neat

[thorn badger]

I will definitely have the serde wrapper so serde_json will work out of the box

[whole berry]

The main reason I would want to use this is to try to get rid of serde but I'll take a compat layer until it gets fleshed out

[thorn badger]

Im currently finishing up how structs are defined with the protocols, and next I need to do enums

[thorn badger]

I might have a super simple one like the serde demo

maybe

[full orbit]

How exactly does the serde wrapper works?
And why would someone choose to use the serde wrapper instead of using serde directly?

And how much overhead would the wrapper add?

[thorn badger]

It wraps any type that implements Deserialize, Serialize, Deserializer, Serializer and makes it implement Build or Walk. This allows any serde compatible type to be composed in a type using treaty. It also allows any treaty enabled type to be used with a serde format

At alpha it's unlikely treaty will have a convincing "use this over serde" for the average user. Except for maybe untagged enums which treaty has better handling of

It does completely replace https://docs.rs/serde-transcode/latest/serde_transcode/

[thorn badger]

I haven't tested it extensively yet, but it should be a minimal overhead as treaty can model serde's data model basically one to one

[last furnace]

That’s so cool

[thorn badger]

Still need to finish the enum representation

[full orbit]

and what this aim to achieve in later versions when its more mature (other then being backward compatible with serde) I mean what will it add?

[thorn badger]

The main aspects are:

• it allows formats basically unlimited freedom (toml can have its date/time be first class, xml can have attributes and namespaces, yaml can have cyclic references, ...),
• it provides full support for untagged and flattened types (instead of serde's half working hacks),
• it allows Value to custom type and back directly (no more extra Serializer and Deserializer),
• it allows async serialization sinks and deserializtion sources (so Json over http can be done on the stream instead of loading it all then doing it),
• provides modes, which allows types to change their behavior when used with treaty (think using camelCase for xml and snake_case for Json, or leaving out fields when sending a http response but having them when talking to your database)

[full orbit]

wow, that's something I can get behind then, I'm especially excited for the async and stream support... this is really missing in serde...
I'm now even more excited for it...

[thorn badger]

Getting async was one of the hardest parts

That's why the effects system now exists

[full orbit]

tbh its really a good thing that you created treaty with serde support in mind... serde is too big and used in rust ecosystem to be replaced anytime soon...

[thorn badger]

Yeah I don't expect someone to convert their whole codebase over to treaty, it's unlikely treaty will ever actually make serde go away if it becomes popular (still a big if)

I may have all the pieces of this worked out, but we don't know if it will result in a cohesive whole when put together

I also want to implement functional lenses with treaty but that's a stretch goal

[thorn badger]

Oh also better errors, I have focused a lot on even no_std environments having full context errors

[last furnace]

frog has a long chunk of cursed code he was posting about

to try and get the async stuff working

[thorn badger]

I only broke rustc like 3 times with it

(and rust analyzer still doesn't understand it)

[full orbit]

tbh with what I see so far its so likely...
Serde was (and is?) good but old, and its hard to add/change new things to old things, because then so many things break...
but this is not the thing for treaty... it can change as much as it want and thats a good thing that we need

[thorn badger]

Yep

[modern jetty]

it's not like serde is hard to modify, it's that the developer doesn't accept PRs

lol

[last furnace]

treaty is a growing child, we can spew whatever toxic propaganda we want into it's head

[modern jetty]

just absolutely no response, for 6 months https://github.com/serde-rs/json/pull/1102

[thorn badger]

The question is can you out do my influence

[last furnace]

no, you are the head propagandist/torturer

maybe I should abandon this metaphor

[full orbit]

Now that you mentioned async
if I remember correctly treaty aimed to support both sync and async... how are you doing that and keeping the sync and async implementation in "sync"? I mean I seen so many libraries struggle with having both sync and async...

[last furnace]

frog show the effects system show the effects system

[thorn badger]

Guy might have their eyes fall out

[last furnace]

worthwhile

[thorn badger]

I have a system to allow async and sync code to be the same

[full orbit]

how exactly are you doing code generation magic?

[modern jetty]

no codegen, just generics

[thorn badger]

you ready for your eyes to bleed?

[last furnace]

just generics is definitely one way of describing it

[modern jetty]

:3

[last furnace]

this bullet in my foot is just a rock

[full orbit]

born ready

[thorn badger]

fn new_walk<'a: 'c, 'b: 'c, 'c>(
    &'a mut self,
    visitor: DynVisitor<'b, 'ctx>,
) -> ErasedEffective<'c, Status, E> {
    // Reset the errors to default state.
    self.error = None;

    // Reset the field index to the default.
    self.index = 0;

    E::as_ctx((self, visitor), |(this, visitor)| {
        visit_request_hint::<E>(visitor.cast(), DynWalker(*this))
            .map(VisitResult::unit_skipped)
            .cast()
    })
    .if_not_finished(|(this, visitor)| {
        visit_value::<_, E>(visitor.cast(), BorrowedStatic(this.value))
            .map(VisitResult::unit_skipped)
            .cast()
    })
    .if_not_finished(|(this, visitor)| {
        visit_tag::<TagConst<{ TAG_TYPE_ID.to_int() }>, E, _>(
            TagConst,
            visitor.cast(),
            ValueWalker::new(TypeId::of::<I::T>()),
        )
        .map(|result| this.record_tag_error(result).unit_skipped())
        .cast()
    })
    .if_not_finished(|(this, visitor)| {
        visit_tag::<TagConst<{ TAG_STRUCT.to_int() }>, E, _>(
            TagConst,
            visitor.cast(),
            NoopWalker::new(),
        )
        .map(|result| this.record_tag_error(result).unit_skipped())
        .cast()
    })
    .if_skipped(|(this, visitor)| {
        visit_tag::<TagConst<{ TAG_MAP.to_int() }>, E, _>(
            TagConst,
            visitor.cast(),
            NoopWalker::new(),
        )
        .map(|result| this.record_tag_error(result).unit_skipped())
        .cast()
    })
    .if_not_finished(|(this, visitor)| {
        visit_sequence::<E>(visitor.cast(), *this)
            .map(VisitResult::unit_skipped)
            .cast()
    })
    .remove_ctx()
    .map(VisitResult::to_status)
}

there are 6 yield points here

[last furnace]

boom boom cat boom boom cat if_not_finished if_not_finished if_not_finished

[last furnace]

but frog! I see no await calls

also how does this work at all

[thorn badger]

want to see the function that makes this all work?

fn r#do<
    'ctx: 'lt,
    'wrap,
    Pre,
    Ctx,
    Owned,
    First,
    FirstOutput,
    FirstPost,
    Done,
    Extra,
    Repeat,
    RepeatOutput,
    RepeatPost,
    Post,
    Return,
>(
    self,
    pre: Pre,
    first: First,
    first_post: FirstPost,
    repeat: Repeat,
    repeat_post: RepeatPost,
    post: Post,
) -> ErasedEffective<'wrap, Return, Self::Effect>
where
    Pre: Ss + FnOnce(Self::Output) -> (Ctx, ControlFlow<Done, Owned>),
    First: Ss
        + for<'temp> FnOnce(
            &'temp mut Ctx,
            Owned,
        )
            -> ErasedEffective<'temp, FirstOutput, Self::Effect, &'wrap ()>,
    FirstPost: Ss + for<'temp> FnOnce(&'temp mut Ctx, FirstOutput) -> ControlFlow<Done, Extra>,
    Repeat: Ss
        + for<'temp> FnMut(
            &'temp mut Ctx,
            &'temp mut Extra,
        )
            -> ErasedEffective<'temp, RepeatOutput, Self::Effect, &'wrap ()>,
    RepeatPost: Ss
        + for<'temp> FnMut(&'temp mut Ctx, &'temp mut Extra, RepeatOutput) -> ControlFlow<Done>,
    Post: Ss + FnOnce(Ctx, Option<Extra>, Done) -> Return,
    Return: Ss,
    RepeatOutput: Ss,
    FirstOutput: Ss,
    'lt: 'wrap;

thats the signature

[last furnace]

how does this work at all again

I don't understand how this is possible

[thorn badger]

in async mode it uses async blocks to make self referential types, in sync mode it doesn't need them

[full orbit]

what in dark magic...
seems like you are aiming to work on the black magic part of project alone forever

[thorn badger]

thing is I think all the black magic is done

I havent ran into any other issues so far

[modern jetty]

I absolutely can't wait to see the code size implications of all those generics

[last furnace]

the all seeing eye of llvm sees through all your layers of indirection

hopefully

[thorn badger]

its actually the same as if it was hand written, I have checked a lot (minus when llvm gets a little confused)

[modern jetty]

yes, 2000 times as it has to compile that function post-mono lol

[thorn badger]

the function just abstracts the notion of borrowing a value while in a loop

thats all it does

let (ctx, done, extra) = match pre(self.0) {
    (mut ctx, ControlFlow::Continue(owned)) => {
        let first_output = first(&mut ctx, owned).0;

        let (done, extra) = match first_post(&mut ctx, first_output) {
            ControlFlow::Continue(mut extra) => loop {
                let repeat_output = repeat(&mut ctx, &mut extra).0;

                match repeat_post(&mut ctx, &mut extra, repeat_output) {
                    ControlFlow::Continue(()) => {}
                    ControlFlow::Break(done) => break (done, Some(extra)),
                }
            },
            ControlFlow::Break(done) => (done, None),
        };

        (ctx, done, extra)
    }
    (ctx, ControlFlow::Break(done)) => (ctx, done, None),
};

Value(post(ctx, extra, done), Default::default())

is the sync impl

[thorn badger]

now the compilation times...

no comment

[full orbit]

if this is the signature Im scared of the function itself
on a unrelated not you must have a lot of fun with rustfmt playing with your code

[thorn badger]

the impl is actually tiny, but the signature is a monster

[thorn badger]

I have given up with this code formatting in a meaningful way

[last furnace]

rustfmt not built for the normal usecase of having 13 generic arguments and approximately 15 million where clauses

[thorn badger]

this reminds me I haven't actually moved the async context to this new do function yet

I should do that

[modern jetty]

honestly who needs async syntax and compiler support

if_not_finished

[thorn badger]

fn then<'wrap, U, F>(self, f: F) -> ErasedEffective<'wrap, U, Self::Effect>
where
    F: FnOnce(Self::Output) -> ErasedEffective<'wrap, U, Self::Effect>,
    U: Ss,
    F: Ss,
    'lt: 'wrap,
{
    self.r#do(
        |v| ((), ControlFlow::Continue(v)),
        |_, v| f(v).cast(),
        |_, v| ControlFlow::<_, Never>::Break(v),
        |_, _| unreachable!(),
        |_, _, _: Never| unreachable!(),
        |_, _, v| v,
    )
}

here is the Future::then function written in the effects system with the do operator

[modern jetty]

frog will just do it for you

[last furnace]

did you have to call it do

[thorn badger]

and yes Never is the real never type

[last furnace]

based

[thorn badger]

I don't want anyone sane to ever call it

[last furnace]

fair

I won't point out the obvious implication of what that says about you calling it

[full orbit]

time to create the rustfmt.toml and tell it what to do... (still going to make the ugly tho, just a little less)

[thorn badger]

fn if_not_finished<'ctx, 'wrap, Ctx, F>(
    self,
    f: F,
) -> ErasedEffective<'wrap, (Ctx, VisitResult<()>), Self::Effect>
where
    Self: Effective<'lt, Output = (Ctx, VisitResult<()>)>,
    F: for<'temp> FnOnce(
        &'temp mut Ctx,
    )
        -> ErasedEffective<'temp, VisitResult<()>, Self::Effect, &'ctx ()>,
    'ctx: 'lt,
    Ctx: Ss,
    F: Ss,
    'lt: 'wrap,
{
    self.r#do(
        |(ctx, v)| {
            (
                ctx,
                match v {
                    VisitResult::Skipped(()) | VisitResult::Control(Flow::Continue) => {
                        ControlFlow::Continue(())
                    }
                    v => ControlFlow::Break(v),
                },
            )
        },
        |ctx, ()| f(ctx).cast(),
        |_, v| ControlFlow::<_, Never>::Break(v),
        |_, _| unreachable!(),
        |_, _, _: Never| unreachable!(),
        |ctx, _, v| (ctx, v),
    )
}

[last furnace]

somedays I wake up feeling Effective but other days I wake up feeling ErasedEffective

[thorn badger]

literally every Future adapter can be written with r#do (minus join which I separate out)

[last furnace]

that's kinda amazing

can we have a Frog Thoughts post about a simplier example of how this mixing async and sync works

[thorn badger]

... yes

[thorn mantle]

what i am curious about is when one would want an async serializer 🤔

[thorn badger]

think reqwest with a json body

or well thats the other way

but in treaty both the serialize and deserialize are the same operation

[thorn mantle]

so like deserializing a body as it comes in?

[thorn badger]

in this context its "the source byte stream is serialized into a generic struct"

[thorn mantle]

yeah ok

[full orbit]

imagine leaving all this dark magic stuff for a few months and then coming back to it...
and I can see RA saying you are on your own, im out...
but in the end it worth the result... having a good async/sync support is much needed in something like treaty

[thorn badger]

treaty doesn't have serde's (in my opinion wrong) distinction between serialization and deserialization

[thorn badger]

The one big compromise I had to make was everything treaty uses needs to be Send + Sync

otherwise the async support wouldn't mean anything because it wouldn't work with tokio and other multithreaded runtimes

[full orbit]

I mean it does make sesne because of async support...

[last furnace]

so wait what does this actually mean

like how can you not have that distinction

[thorn badger]

in treaty everything is always the walker -> builder flow, data flows from a walker to a builder

[last furnace]

hmm

[thorn badger]

mapping it to serde its Deserializer/Serialize = Walker, Deserialize/Serializer = Builder

[full orbit]

how it work both side?

[thorn badger]

so when serializing a value to json we walk the Rust value and build a sequence of text that is the json value, when we deserialize a json strict to a Rust value we iterate over the json string and build the rust value

in reality formats are the same thing as any other data type

for example serde_json's Value type

which you can serialize into and deserialize from

[thorn mantle]

so treaty would let you convert between 2 rust values automatically right?

[thorn badger]

yep

pub fn transform<'a, 'ctx: 'a, B: Builder<'ctx, E> + 'a, W: Walker<'ctx, E> + 'a, E: Effect>(
    seed: B::Seed,
    walker: W,
) -> ErasedEffective<'a, (Result<B::Value, B::Error>, Result<W::Output, W::Error>), E> {
    B::from_seed(seed)
        .as_ctx(|builder| walker.walk(builder.as_visitor()).cast())
        .then(|(builder, walker_result)| {
            builder
                .build()
                .map(|builder_result| (builder_result, walker_result))
        })
}

this operation allows doing things like converting one struct into another, and json to toml

along with struct to json, and json to struct

This was one of the foundational ideas of treaty I designed in from looking at discussions around serde

We simplify the API surface to 1 abstraction instead of 2

the abstraction being getting data in format A into format B where A and B don't know about each other beforehand

[full orbit]

so on the ground treaty should perform much faster in both serialize and deserialize then serde?

[thorn badger]

thats to be seen, I have done a lot of testing to make sure the code gen is as good as possible, so it should be no slower than serde in the general case

[forest osprey]

you're taking the "everything is a file" approach to another level:
"everything is a data format, including rust data types"

[thorn badger]

I haven't been able to benchmark against serde yet as I don't have enough types for it to be a fair comparison

[thorn badger]

if we are going to have a data model might as well use the same one (minus that treaty doesn't have a data model, but a "protocol set and flows")

hence the name treaty, format A and B work together to find what protocols they know about to allow passing data from A to B

What we define to allow this to work is a flow for what to try in what order

[thorn badger]

protocols here in the base treaty are passing a value (any* type), and iterating over a sequence, and tagging a value with meta information

I do admit this makes writing a maximally compatible serialize/deserializer challenging compared to serde's data model

[full orbit]

it would be nice to see some benchmark when possible...
btw one unrelated question
serde for example doesn't use const generic and because of that it can only (de)serialize lists with a fixed size that are already are predefined in code (just because [T; 0] I think)...
does treaty also like this?

[thorn badger]

treaty allows arbitrary types to flow between walker and builder, so if the walker has a [T; N] and the builder wants to work with it then thats allowed automatically

impl<'ctx, E: Effect> Value<'ctx, OwnedStatic<bool>, E> for Builder<E> {
    fn visit<'a>(
        &'a mut self,
        OwnedStatic(value): OwnedStatic<bool>,
    ) -> ErasedEffective<'a, VisitResult<OwnedStatic<bool>>, E>
    where
        'ctx: 'a,
    {
        self.0 = Some(value);
        E::ready(Flow::Done.into())
    }
}

here is an example from the code as it is now showing how a bool value is transferred to a builder

[last furnace]

so now all you need is a big chunk of examples to convince people away from serde

[thorn badger]

yeah that will happen eventually

[last furnace]

what's an owned static

[thorn badger]

A wrapper forced because I gave lifetime containing types TypeId's, so a OwnedStatic<T> allows any T: 'static to be nameable using this system

[full orbit]

neat, I better stop asking question, because by each answer I want to use treaty more and more
this also was a pretty big limitation of serde to me...

[thorn badger]

I also have ones like pub struct BorrowedStatic<'ctx, T: ?Sized>(pub &'ctx T);

[last furnace]

scary

that's kinda cool

[last furnace]

it does sound pretty sick

[thorn badger]

yep, treaty is built with no given data model, you get to choose whatever you want (minus the Send + Sync and only 1 lifetime)

The past few months have been "is all this possible without compromising one of the other main features"

[full orbit]

only one lifetime?

[thorn badger]

and at this point the answer is yes

[thorn badger]

like how serde has only the 'de lifetime

I refrained from going with more than one context lifetime

so a type like

pub Demo<'a, 'b> {
  a: &'a str,
  b: &'b str,
}

isn't going to be supported (assuming we dont use Demo<'a, 'a>)

[full orbit]

to be honest Im still wondering how all this is possible... its very interesting what you achieved
any plan for a release so we could test it a little?

[thorn badger]

If I could pass around lifetime like I do types then it would be possible, but alas we cant

[thorn badger]

I am aiming for end of the month for the alpha release

it really depends on how much free time I end up with after work

[last furnace]

that's pog

[thorn badger]

This includes full (or at least mostly) docs and examples for what is there

and probably a descent into madness post about the effects system since a lot of people have wanted me to do that

[full orbit]

full doc from start in a alpha release who heard of that

[thorn badger]

I already have a lot of them, considering even I would probably go insane having to return to treaty after a week away with no docs

[last furnace]

lol

[thorn badger]

Also it would be very bad to promise all these things and then I never actually tested they work together (hence the last few months of convincing rustc its ok)

[sharp condor]

https://tenor.com/view/cat-cats-explosion-explodes-cat-explodes-gif-10311420692458175149

[last furnace]

me when treaty releases

[thorn badger]

Oh also I had a goal of minimal unsafe, which I think I have accomplished. I only need unsafe in 1 spot (2 depending on how you count it)

and thats for the lifetime containing downcasting

because I already have to worry about the higher ranked soundness hole, worrying about a treaty sized block of unsafe code would be

[thorn mantle]

miri catches it when hrtb fucks up right?

[thorn badger]

assuming you actually trigger UB with it

I think I have made it unlikely for a user to accidentally cause it with treaty's API, but im also not 100% sure how I would even quantify that in practice

Like I have never managed to do it by accident so far even though its very close to the examples

[full orbit]

no forbid[unsafe_code] then

[thorn badger]

and to my knowledge we have no static analysis tools to check for the soundness hole

[thorn badger]

If i removed that piece of code to a separate crate we could

its basically a more fancy version of https://docs.rs/better_any/latest/better_any/

I am 95% sure thats the only spot that uses unsafe

If I do that I would probably also spin off the const symbols

[next tundra]

rg unsafe

[last furnace]

one easy trick to make your crate unsafe free!

[thorn badger]

and probably the effects

[full orbit]

we could have a even_better_any

[thorn badger]

but we can do that after the alpha

[next tundra]

betterer any

[sharp condor]

same

[thorn badger]

https://marketplace.visualstudio.com/items?itemName=tamasfe.even-better-toml moment

[full orbit]

exactly what was in my mind when I said it 😂

[thorn badger]

https://gitlab.com/konnorandrews/treaty/-/blob/main/src/symbol.rs?ref_type=heads#L903 is probably my single favorite file in treaty

[full orbit]

why tho

[thorn badger]

because I made it to one up sam's crate

https://docs.rs/smol-symbol/0.1.7/smol_symbol/

[last furnace]

what is this

[thorn badger]

and the giant lookup table makes me smile every time

[last furnace]

oh this is just arithemetic coding

lol

[thorn mantle]

What the fuck lol

[thorn badger]

the fun part is I couldn't use floats for it

[last furnace]

why not

are these just ratios

[thorn mantle]

Const limitations?

[thorn badger]

because const

[last furnace]

oh lol

[full orbit]

its actually interesting, but only for english alphabet? although no one in right mind would use other character

[thorn badger]

yeah otherwise each language would get like 1 symbol to use

because the whole string has to fit into the space of 2 utf32 chars

also I once again dislike that we can't use strings for const generics

[last furnace]

yeah

kinda sucks

[full orbit]

what is the main difference compared to smol-symbol? seems like both try to achieve same thing?

[full orbit]

would be nice if it was possible...

[thorn badger]

mine is a u64 vs sam's u128 but with a similar max size of string

also mine allows a few more characters than his (assuming you don't use a custom alphabet)

[full orbit]

what could be the max size of string be in your version?

[thorn badger]

with the current model it maxes at 16

https://gitlab.com/konnorandrews/treaty/-/blob/main/src/symbol.rs?ref_type=heads#L721 shows examples

The goal was to allow most "programming" terms/names

sam's crate uses a fixed width encoding

[full orbit]

interesting... this could kinda a be a separate crate...
in treaty usecase we know the input string never get bigger then 16?

[thorn badger]

I use these as types any crate can define and name

So if crate A uses ConstTag<{ Symbol::new("My Name").to_int() }> and crate B also uses that then they are the same

this is used by treaty's tag protocol to add meta information

This is to allow for some external source to define a new tag and crates don't have to depend on one definition crate to use it

[full orbit]

oh ok, I haven't seen much of treaty Tag system so Im not sure how it works, but this seem interesting

[thorn badger]

visit_tag::<TagConst<{ TAG_STRUCT.to_int() }>, E, _>(
    TagConst, // needed because runtime tags are also allowed
    visitor.cast(), // the builder to try and use the tag with
    NoopWalker::new(), // the value associated with the tag (here its an empty value for this tag)
)

here is a basic example of it being used

this tells the builder the walker has a struct to send it

visit_tag::<TagConst<{ TAG_TYPE_ID.to_int() }>, E, _>(
    TagConst,
    visitor.cast(),
    ValueWalker::new(TypeId::of::<I::T>()),
)

here is one with a value

[full orbit]

so tag are used by visitor to know which walker to use for it?

[thorn badger]

can be

they are also used for things like keys in a map

the key is a tag

visit_tag::<tags::Key, E, _>(TagConst, visitor.cast(), key_walker)

so for a map/struct it does

visit map/struct tag
for field:
  visit key tag
    visit key value
  visit field value

[full orbit]

got it
the overall system is so complex ngl, but how it work all together is so interesting

does serde have something similar to treaty tags?

[thorn badger]

yep, I am probably going to need to make some diagrams to help people understand how the flows work

[thorn badger]

no

what people usually do instead in serde is magic field names

even serde does this internally

[full orbit]

yeah that would be nice

[thorn badger]

by the way thanks for asking questions, each time i explain it, it becomes easier to explain it next time

[full orbit]

no thank you for answering them... I tend to sometimes ask so many questions 😅

[modern bone]

So in the process of trying to develop my own lib, the main problem I found myself running into is nesting structures. For example, if you want to serialize an array as a field, that array needs to ask the object collector for a list collector. Is this where IDETs come in to treaty, to allow you to try and treat a given Visitor in a certain way?

[thorn badger]

yes

[thorn badger]

In treaty this specific form is done here ```rs
pub fn visit_sequence<'a, 'ctx, E: Effect>(
visitor: DynVisitor<'a, 'ctx>,
scope: DynSequenceScope<'a, 'ctx, E>,
) -> ErasedEffective<'a, VisitResult<DynSequenceScope<'a, 'ctx, E>>, E> {
if let Some(object) = visitor.0.upcast_mut::<SequenceProto<E>>() {
// Allow the visitor to walk the sequence scope.
object.visit(scope)
} else {
// If the visitor doesn't support sequence then we continue.
VisitResult::Skipped(scope).ready()
}
}

DynSequenceScope here is basically Iterator

pub trait SequenceScope<'ctx, E: Effect> {
    fn size_hint(&mut self) -> ErasedEffective<'_, (usize, Option<usize>), E>;

    fn next<'a: 'c, 'b: 'c, 'c>(
        &'a mut self,
        visitor: DynVisitor<'b, 'ctx>,
    ) -> ErasedEffective<'c, Flow, E>
    where
        'ctx: 'c + 'a + 'b;
}

the visitor.0.upcast_mut::<SequenceProto<E>>() is where the DIDET is

[thorn badger]

this makes visitor into a &mut dyn Sequence<'ctx, E> with ```rs
pub trait Sequence<'ctx, E: Effect> {
fn visit<'a: 'c, 'b: 'c, 'c>(
&'a mut self,
scope: DynSequenceScope<'b, 'ctx, E>,
) -> ErasedEffective<'c, VisitResult<DynSequenceScope<'b, 'ctx, E>>, E>
where
'ctx: 'a;
}

[modern bone]

Interesting

[last furnace]

continue doing it

we must fuel frog

[visual tartan]

O is it coming out

[visual tartan]

What

Why effect system thing

[thorn badger]

do you have another recommendation? I currently dont have anything really planned for them

[visual tartan]

Ah do you mean just to meld in async with sync?

If so, I see.

Effect system has connotation of handling all sorts of things ime

[thorn badger]

the effects system allows code to be generic over async or blocking

so not the programming theory effects

[visual tartan]

Yeah

Would be sucky to implement..

Borrow checker will hate that

[thorn mantle]

will you have an effect for coroutines

[thorn badger]

it supports yielding with no value

thats it

technically you probably could get coroutines to work out

I don't think the do function forbids them

you know the do function probably allows quite a few effects now that I think about it ...

I already have fallibility

[last furnace]

[thorn badger]

const is definitely not one of them

and I have repetition...

great I have a full alternative to conrad's effective apparently

[last furnace]

That moment when you accidentally write a full effects system when all you want to do is serialise json

[thorn badger]

The thought: what if it was like javascript promises before async await
The outcome: full effects system

[thorn mantle]

surely someday rust will have its own effects system

[last furnace]

Any sufficiently complicated Rust program contains an ad hoc, informally-specified, bug-ridden, slow implementation of an effects system

[visual tartan]

What

[thorn badger]

https://doc.rust-lang.org/std/task/enum.Poll.html#variant.Pending

[visual tartan]

Why and how

[thorn badger]

do you want your eyes to melt?

[visual tartan]

Yeah I am fine with that

[thorn badger]

@visual tartan ^

[visual tartan]

The hell?

[thorn badger]

welcome to my effects system in stable rust

[visual tartan]

Devil magic

What happened to result in this monstrosity?

[thorn badger]

result?

[visual tartan]

I mean

How did this monstrosity come to being

[thorn badger]

ah

[visual tartan]

FirstPost RepeatPost

[thorn badger]

I wanted to support async but without needing everything to be futures

[visual tartan]

Is it doing event handling here?

[thorn badger]

thats abstracted away

[visual tartan]

Wha, how

[thorn badger]

in async mode ErasedEffective becomes a Box<dyn Future>

[visual tartan]

So do you handle the difference of sync/async with some kind of event handling logic?

I bet everything as future could have been easier

[thorn badger]

no, thats the route conrad's effective crate goes, I went a different route

in my system you write everything as blocking code

and when its about to yield the scope ends

[visual tartan]

But how does this work

[thorn badger]

all the async stuff happens between closure scopes

[visual tartan]

Like, blocking code is fundamentally different from nonblocking code, right

[visual tartan]

How do you detect closures

[thorn badger]

if you cut nonblocking into many pieces then its blocking again

[visual tartan]

Oh so you wrote entire async framework

[thorn badger]

thats on the developer to do

[thorn badger]

no it still needs a separate runtime of your choice

[visual tartan]

Yeah but still

[thorn badger]

but in some sense yes, yes I did

all based on one function

namely r#do

[visual tartan]

You are converting some blocking-lookalike-code automatically into nonblocking one

Devil magic really

[thorn badger]

E::as_ctx((self, visitor), |(this, visitor)| {
    visit_request_hint::<E>(visitor.cast(), DynWalker(*this))
        .map(VisitResult::unit_skipped)
        .cast()
})
.if_not_finished(|(this, visitor)| {
    visit_value::<_, E>(visitor.cast(), BorrowedStatic(this.value))
        .map(VisitResult::unit_skipped)
        .cast()
})
.if_not_finished(|(this, visitor)| {
    visit_tag::<TagConst<{ TAG_TYPE_ID.to_int() }>, E, _>(
        TagConst,
        visitor.cast(),
        ValueWalker::new(TypeId::of::<I::T>()),
    )
    .map(|result| this.record_tag_error(result).unit_skipped())
    .cast()
})
.if_not_finished(|(this, visitor)| {
    visit_tag::<TagConst<{ TAG_STRUCT.to_int() }>, E, _>(
        TagConst,
        visitor.cast(),
        NoopWalker::new(),
    )
    .map(|result| this.record_tag_error(result).unit_skipped())
    .cast()
})
.if_skipped(|(this, visitor)| {
    visit_tag::<TagConst<{ TAG_MAP.to_int() }>, E, _>(
        TagConst,
        visitor.cast(),
        NoopWalker::new(),
    )
    .map(|result| this.record_tag_error(result).unit_skipped())
    .cast()
})
.if_not_finished(|(this, visitor)| {
    visit_sequence::<E>(visitor.cast(), *this)
        .map(VisitResult::unit_skipped)
        .cast()
})
.remove_ctx()
.map(VisitResult::to_status)

here is some "async" code

[last furnace]

Ctfl f “await” 0 results found

Funky

Hopefully we’ll get plenty of Frog Thoughts about however the fuck this works

[visual tartan]

I am having difficulty how this chains

[thorn badger]

fun thing is this works in the before async time

[visual tartan]

Parens are hard

[thorn badger]

dont worry it hurts my brain also

and I wrote the stupid thing

[visual tartan]

There does seem to be tons of workers doing stuff

And maps, no idea what they do

[visual tartan]

Btw what's with all the tags and hints?

[thorn badger]

part of treaty's internals

its how pieces talk to eachother

[visual tartan]

So it's not going to be exposed?

[thorn badger]

it will be, but only for those making custom formats

[visual tartan]

Ah

[last furnace]

So how hard would it be to make one for a niche format, like json? Or a more mainstream format, like json?

Or even a weird format like json

[thorn badger]

once I have all the helpers probably not bad, the real complexity would be if you want something brand new in the treaty "ecosystem"

like handling a graph

[visual tartan]

Can it parse XML

[last furnace]

you'd still have to write an xml parser

[visual tartan]

F

[spring folio]

can't you do like pre: impl Ss + FnOnce... so it looks better?

[thorn badger]

I don't like mixing generics like that

But maybe

[spring folio]

rustfmt may unexplode if you do that xd

also i see you have a lot of unreachable!()s, I may do a mr where i just inline the shit out of the do's (by hand.)

also is there a derive yet? (if there isnt i can write one)

[thorn badger]

This is the version where I abstracted all of them to do

[spring folio]

oh

[thorn badger]

I have basic macro rules ones

[spring folio]

I know derives so once i can i will do a merge req

[thorn badger]

Which is what I plan on having for the alpha

[thorn badger]

Feel free if you want to

Just no guarantees I will accept it as is

[spring folio]

absolutely expected, i dont even know what you're aiming for for the syntax

if you can tell me any ideas that would be 👍

[thorn badger]

Having each context implement each adapter was extremely painful and didn't allow for easy custom adapters

[spring folio]

#[inline(always)]

[thorn badger]

Yep

It's slightly sad that llvm messes up eventually as it starts running out of virtual registers when inlining so has to stop some of its optimizations

So instead of perfect code gen you get a few extra branches

[spring folio]

inline asm

[thorn badger]

No unsafe

[spring folio]

frog: perfect codegen
also frog: no unsafe

wait no you can do a feature flag with inline mir

naming idea: rename .r#do to .magic or .explode (||JOKE||)

[visual tartan]

Oh how heavy would this effect be

[silver bolt]

eh idk

[silver bolt]

to quote nora:
sometimes the "unsafer" thing is better than the "safer" thing

[silver bolt]

also hold on-- you have unsafe?

[silver bolt]

inline asm isnt very good though

inline mir isnt very good either

[spring folio]

yeah its platform specific (=pain)

[silver bolt]

thats not the problem

the problem is that this code works with functions

inline asm doesnt fix function inlining

[spring folio]

oh yes

[full orbit]

Too much fuel could lead to an explosion, I'm scared of that

[thorn badger]

I am predicting a minimal extra overhead

[visual tartan]

Ah thats good to hear

[quaint raft]

frog has achieved maximum job security

there can't be many people capable of that much Rust black magic

[digital gazelle]

Wondering how much of a size difference it will make versus serde (and miniserde)

[thorn badger]

enum X {
    A(f32),
    B(bool),
}

let mut builder = X::new_builder();

// Use the recoverable flow.
{
    let mut scope = MockRecoverableScopeVisitor::<Blocking>::new();

    // The first builder for a f32 won't work.
    // In a real walker the scope would always walk the same.
    scope.expect_new_walk().once().return_const(Status::Ok);

    // The second builder will work.
    scope.expect_new_walk().once().returning(|visitor| {
        // The value for the B variant.
        visitor.visit_value_and_done(OwnedStatic(true));

        Status::Ok
    });

    // Visit a recoverable scope the enum builder can use to try and find
    // a variant that can be built.
    assert_eq!(
        visit_recoverable(builder.as_visitor(), &mut scope).value(),
        Flow::Done.into()
    );
}

// The enum should have a value now.
assert_eq!(builder.build().value().unwrap(), X::B(true));

Untagged enums work (it tries each variant's builder until one works)

[last furnace]

woh

[thorn badger]

let mut builder = X::new_builder();

assert_eq!(
    visit_tag::<tags::Variant, Blocking, _>(
        TagConst,
        builder.as_visitor(),
        ValueWalker::new(0u32),
    )
    .value()
    .unwrap(),
    Flow::Done.into()
);

builder
    .as_visitor()
    .visit_value_and_done(OwnedStatic(1.23f32));

assert_eq!(builder.build().value().unwrap(), X::A(1.23));

Giving a tag also works now

fn guess_variant<'a>(
    _seed: Self::Seed,
    scope: DynRecoverableScope<'a, 'ctx, E>,
) -> ErasedEffective<'a, Result<Self::T, Self::Error>, E> {
    E::as_ctx(scope, |scope| {
        <<f32 as Build<M, E>>::Builder as Builder<_>>::from_seed(Default::default())
            .map(|builder| (scope, builder))
            .as_ctx(|(scope, builder)| scope.new_walk(builder.as_visitor()).cast())
            .then(|((_, builder), result)| builder.build())
            .map(|result| result.map(X::A))
            .cast()
    })
    .as_ctx(|(scope, result)| {
        <<bool as Build<M, E>>::Builder as Builder<_>>::from_seed(Default::default())
            .map(|builder| (scope, builder))
            .as_ctx(|(scope, builder)| scope.new_walk(builder.as_visitor()).cast())
            .then(|((_, builder), result)| builder.build())
            .map(|result| result.map(X::B))
            .cast()
    })
    .map(|((scope, _), result)| match result {
        Ok(value) => Ok(value),
        Err(err) => todo!("{}", err),
    })
}

the internals of the guessing hurts my soul

[full orbit]

The concept of guessing itself hurt my soul but it is what it is

[nimble dawn]

I just read about half of this thread and my head is spinning

[dense oasis]

The recap is "Frog has made a library that amounts to impl TryFrom<A> for B, except A and B don't have to know about each other".

The internals are, uh, complicated. My understanding is that the surface API is less incomprehensible

Or at least, will be, when it comes out.

[nimble dawn]

interesting

from what I've read there is some fascinating (and horrifying) stuff in the internals

the async/sync effect abstraction in particular looks quite interesting

[silver bolt]

methinks you need more option helpers

also that unwrap written with match is funny

[whole berry]

less incomprehensible
so there’s a chance it’s still horrors beyond human comprehension?

[whole berry]

we need a .unwrap_todo ngl

[dense oasis]

We have it, it's expect("todo")

[thorn mantle]

that doesn't get caught by warn(clippy::todos) tho

[visual tartan]

Tbh I expect the internals to be rather messy, as you have to go around typesystem

Otherwise it is like, rather, chain of

impl TryFrom<A> for SomeKnownFoo
impl TryFrom<SomeKnownFoo> for B

or sth

[thorn badger]

It's even more complicated because treaty doesn't use a set data model

I think this quote is going to describe treaty's docs very well

I am going to give what I will call an elementary demonstration. But elementary does not mean easy to understand. Elementary means that very little is required to know ahead of time in order to understand it, except to have an infinite amount of intelligence. --Richard Feynman

[visual tartan]

I recall that it goes through some kind of translation to known types (which could be optimized out)

[wraith sequoia]

just drop an example that isn't 60 lines and I think we will begin to understand

[thorn badger]

You are asking a lot there

[modern jetty]

reminder that you are gonna have to get a less than 60 line example, because 99% of users are looking for ```rs
#[derive(serde::Deserialize)]
struct User {
name: String,
email: String,
}

let user: User = serde_json::from_str("...").expect("json deserialization failed");```

[last furnace]

Aren’t most users secretly wanting to be bombarded with the intricate type system abuse that makes treaty work?

[thorn badger]

"usecase is unsupported"

[dense oasis]

The definition of "better" serialization framework seems to have drifted a bit here

[thorn badger]

Who said I couldn't dig up the goal post and move it 300 ft down field

On a serious note I am wondering if I should put my extension methods for walkers and builders into one trait

Maybe I can use a super trait with both of them as children

As ctx I have some extension traits that provide the "normal" API

[modern jetty]

I think having two traits will be confusing, and follow Iterator's example here.

[thorn badger]

What part of iterator should I use as example?

It's kind of more like the Future trait and FutureExt

I guess I could squish everything into truly one trait the bounds would be super weird though

[visual tartan]

I don't think it's like Iterator

[red kettle]

I think that @modern jetty meant that Iterator's methods bound themselves on extension traits.

so that there's a one-stop-shop

however, Iterator only does this because all Iterators are logically quite similar.

it sounds like walking and building are quite possibly very different

[red kettle]

thus, this sounds like the way to go

[open oriole]

:3

[open oriole]

How the progress on this btw

[digital gazelle]

you can't rush perfection

[austere saddle]

just took a look at the sources

I wish I could understand it

One day I will

but not today

[wraith sequoia]

Me af

[wraith sequoia]

ME. AS. FUCK

[south terrace]

@thorn badger will you ever create a blog on the theory behind any.rs? because its making my head hurt to understand it, and i do not want to learn type theory to understand a single file of code.

[thorn badger]

I haven't yet

[thorn badger]

#rust-discussions-1 message

[visual tartan]

Then you will be one of two people who understands it in the world.

[wraith sequoia]

So on a basic level, walkers drive visitors, right?

[thorn badger]

Yes

[full orbit]

Could you give a little more context and how it's related to serde?

[thorn badger]

Serde doesn't restrict access types to T: Send, so I can't either

[wraith sequoia]

So does each value get visited with a different protocol by a different Walk implementation, or does a Walker only receive one DynVisitor that it reuses

[thorn badger]

One DynVisitor that it can reuse

[thorn badger]

Note that if you look at the code you will actually see most of the time the visitor drives the walker do to how the flow works, the walker should ask for a hint as it's first protocol

If the visitor doesn't want to hint anything then the walker proceeds as normal driving the visitor

Also Walk exists to assign a canonical Walker to a type (same for Build)

the core API of treaty doesn't actually care that Walk is a thing, it's used for the derive default and helper methods

[wraith sequoia]

Okay, very good to know

So does every visitor have the opportunity to provide a hint? As in every protocol, in technicality, can carry two protocols?

[thorn badger]

Yep

The hint protocol takes any* other protocol

[wraith sequoia]

So protocols are composable?

[thorn badger]

Yep

Though hint is the only one I have that uses this

[wraith sequoia]

How do you compose them?

[thorn badger]

Using generics, like for hinting a transfer of a value of type T it's Hint<Value<T>> (simplified)

This is an application of treaty's lack of a given data model and protocol set, all the base crate gives are some well agreed upon protocols (abstractions of serde and other crate's abilities)

The base protocols being, Value, Tag, Sequence, Recoverable, and Hint (also RequestHint)

For two parties (walker and visitor) to communicate what we lay out are flows for how they should negotiate data transfer between themselves using some set of protocols in some order

So if a given walker or visitor doesn't respect a specific flow then they are incompatible (at runtime) but that's what we give up in order for the massive flexibility and extendability

Serde suffers from the same issue but doesn't really acknowledge it

For example attempting to deserialize a untagged enum from bincode

In treaty this case still doesn't work, but it's considered a case of neither party having enough information to do a successful data transfer

[dense oasis]

I presume, of course, that it is not practical to have the error come in the form of some "trait bound not satisfied" error?

Honestly I can't complain, treaty does enough that I can put up with an unlikely runtime error that I already put up with in serde 😄

[thorn badger]

Yeah, especially since some walkers and visitors change behavior at runtime

[thorn badger]

I did get introduced to https://github.com/Munksgaard/session-types which could allow some walkers and visitors to agree at compile time what flow they will use

But I'm considering this out of scope for treaty

[wraith sequoia]

Would it be weird of me to make a dumbed down reimplementation of treaty's concepts?

[thorn badger]

Not at all

I'm sure the same concepts could be applied in a much simpler form

My self imposed requirements are quite limiting on the simplicity I can achieve

Basically toss out async generics and lifetime containing types and everything becomes much easier

[wraith sequoia]

First I just have to understand them v.v

Speaking of, why are the protocols implemented as free functions?

[modern bone]

That's what I've been trying to do, but without all the runtime negotiation (partially because it's not comptime, but mostly because i'm not smart enough)

[austere saddle]

what do I need to learn to understand hkt.rs and any.rs(TypeName)? Type theory?

[thorn badger]

They aren't, you are probably thinking of the helper functions that handle using AnyTrait and calling the trait method

[thorn badger]

Higher kinded types, and higher ranked types, specifically what these mean in terms of Rust's type system