Help implementing serde wrapper types for nested serialization behavior | Rust Programming Language Community | Page 1

bronze fiber Jun 28, 2022, 10:14 PM

#

I am having a very hard time figuring out the impls for the crate serde_with, but don't eve know how to ask the question because I am not entirely clear on how it is supposed to work. I have been struggling with this for three or four days straight, reading the implementations of the traits in the crate itself. It still doesn't make sense, and everything I try confuses me more. How do you ask a question when there is no specific question to ask? I wouldn;t even know where to begin with creating a thread for this, I can only mention the goal and ope that someone else is familiar with serde_with. I know this is a relatively hard problem; but I would seriously appreciate some attention for it.

I'm trying to implement a wrapper type for serde that takes a struct's field, in one instance Vec<u64> and serialize with the adapter types JsonString and Base62. The adapters are used with an attribute macro on the field, such as #[serde_as(as = JsonString<Vec<Base62<u64>>>)].

JsonString<T>(PhantomData<T>) and Base62<T>(PhantomData<T>), both need to implement SerializeAs and DeserializeAs. I am getting very confused with the trait bounds and the call stack because it is nested so deep, and these types, especially with Deserializer::deserialize perform mutable operations on data they hold, and I don't now how to organize the interaction with it.

#

To start, we are here:

#[cfg(test)]
pub mod tests {
    use fake::faker::name::en::{FirstName, LastName};
    use fake::{Dummy, Fake};
    use serde::{Deserialize, Serialize};
    use serde_with::serde_as;
    use time::OffsetDateTime;

#

Now, to define a dummy type:

 #[serde_as]
    #[derive(Clone, Debug, PartialEq, Serialize, Deserialize)]
    struct Human {
        first: String,
        last: String,
        #[serde(with = "time::serde::rfc3339")]
        birth: OffsetDateTime,
    }

    impl<F> Dummy<F> for Human {
        fn dummy_with_rng<R: rand::Rng + ?Sized>(_: &F, rng: &mut R) -> Self {
            Human {
                first: FirstName().fake_with_rng(rng),
                last: LastName().fake_with_rng(rng),
                birth: OffsetDateTime::from_unix_timestamp(rng.gen_range(0..253402300799)).unwrap(),
            }
        }
    }

#

This is the first test, with one level of nesting.

    #[test]
    fn test_roundtrip() {
        // Make an "anonymous" struct that annotates the field `values` with the adapter
        // type `JsonString`. The field should first serialize every `Human` as a JSON
        // object-string, and then make a `Vec<String>` and serialize that into JSON
        // when the `TestContainer` is serialized.
        let container = crate::macros::new_struct! {
            #[serde_as]
            #[derive(Serialize, Deserialize)]
            TestContainer {
                #[serde_as(as = "Vec<JsonString<Human>>")]
                pub values: Vec<Human> = fake::vec![Human; 50],
            }
        };
        // The nested serializing should be handled by the `serde_as` procedural macro.
        let serialized = serde_json::to_string(&container).unwrap();
        // Now, deserialize the above example and unwrap the JSON `Value` to make sure
        // that things adhere to the types that are expected.
        let parsed = serde_json::from_str::<serde_json::Value>(&serialized).unwrap();
        let parsed = parsed
            .as_object()
            .unwrap()
            .get("values")
            .unwrap()
            .as_array()
            .unwrap()
            .iter()
            // This is obviously the most significant part, where the nested JSON is manually parsed
            // after getting the `Vec<String>`.
            .map(|v| serde_json::from_str::<Human>(v.as_str().unwrap()).unwrap());

        // Ensure that what was parsed manually matches what was deserialized using the
        // wrapper types defined
        for (expect, actual) in std::iter::zip(container.values, parsed) {
            assert_eq!(&expect, &actual);
        }
    }

#

Now, this is what I actually care about, with the nested types:

    #[test]
    fn test_wrapping() {
        let container = crate::macros::new_struct! {
            #[serde_as]
            #[derive(Serialize, Deserialize)]
            TestContainer {
                #[serde_as(as = "JsonString<Vec<Base62<u64>>>")]
                pub values: Vec<u64> = ((u64::MAX - 1000)..u64::MAX).chain([0]).collect(),
            }
        };
        let serialized = serde_json::to_string(&container).unwrap();
        dbg!(&serialized);

        let parsed = serde_json::from_str::<serde_json::Value>(&serialized).unwrap();
        let parsed = serde_json::from_str::<serde_json::Value>(
            parsed
                .as_object()
                .expect("expected an object map")
                .get("values")
                .expect("expected an object field")
                .as_str()
                .expect("expected a string"),
        )
        .expect("expected to parse string as json");
        let parsed = parsed
            .as_array()
            .expect("expected a vector of strings")
            .iter()
            .map(|v| {
                dbg!(v);
                base62::decode(v.as_str().expect("expected a string as base-62"))
                    .expect("failed to parse base-62") as u64
            });

        for (expect, actual) in std::iter::zip(container.values, parsed) {
            assert_eq!(&expect, &actual);
        }
    }

#

Inside TestContainer you can see #[serde_as(as = "JsonString<Vec<Base62<u64>>>")] is on the field pub values: Vec<u64>.
Serializing should happen as follows:

For every u64 serialize that as a base-62 string.
For every serialized string, collect into a vector.
Serialize that vector as JSON.
Lastly, serialize the TestContainer as JSON again.

Deserializing should follow the reverse process.

#

I have read the documentation for

#

?crate serde_with

subtle quailBOT Jun 28, 2022, 10:36 PM

#

serde_with

Custom de/serialization functions for Rust's serde

Version

1.14.0

Downloads

6 212 569

bronze fiber Jun 28, 2022, 10:37 PM

#

Starting here: https://docs.rs/serde_with/1.14.0/serde_with/guide/index.html

I have also been reading
https://github.com/jonasbb/serde_with/blob/master/serde_with/src/ser/impls.rs
and
https://github.com/jonasbb/serde_with/blob/master/serde_with/src/de/impls.rs

#

The files that matter are here, in my crate:
https://github.com/spikespaz/awaur/tree/master/src/serde_with

#

And here are two discussions on the serde_with crate's github that should sum up my struggles/stupidity:
https://github.com/jonasbb/serde_with/discussions/480
https://github.com/jonasbb/serde_with/discussions/479

bronze fiber Jun 28, 2022, 11:36 PM

#

@quiet oriole

slow slate Jun 29, 2022, 3:14 PM

#

bronze fiber The files that matter are here, in my crate: <https://github.com/spikespaz/awaur...

hey, I managed to get it working, I agree that implementing the *As traits it kinda confusing, but they're flexible enough to get you what you want in the end

I changed 3 things:

Base62 is not a wrapper struct anymore - it's supposed to be always de/serializing a singular value, it's not a container; just removing the <T> makes is simpler
JsonString is a "container", in that you could want to change the inner representation when de/serializing, so it should implement De/SerializeAs in a forwarding manner: it need bounds on De/SerializeAs. The De/SerializeAsWrap types are useful here, as they do that forwarding for you in the implementation. The final impls are like this: rs impl<TAs, T> SerializeAs<T> for JsonString<TAs> where TAs: SerializeAs<T>, { fn serialize_as<S>(source: &T, serializer: S) -> Result<S::Ok, S::Error> where S: Serializer, { super::with::serialize(&SerializeAsWrap::<T, TAs>::new(source), serializer) } } impl<'de, T, TAs> DeserializeAs<'de, T> for JsonString<TAs> where TAs: for<'a> DeserializeAs<'a, T>, { fn deserialize_as<D>(deserializer: D) -> Result<T, D::Error> where D: Deserializer<'de>, { let wrapped: DeserializeAsWrap<T, TAs> = super::with::deserialize(deserializer)?; Ok(wrapped.into_inner()) } } notice the for<'a> DeserializeAs<'a, T>, as it uses the deserialize that needs DeserializeOwned
addressing the requirement of De/SerializeAs instead of De/Serialize you need to use a helper struct, serde_with has Same to just leave the type unchanged: ```rs
let container = crate::macros::new_struct! {
#[serde_as]
#[derive(Serialize, Deserialize, PartialEq)]
TestContainer {
#[serde_as(as = "Vec<JsonString<Same>>")]
pub values: Vec<Human> = fake::vec![Human; 50],
}
};

#

I also added a test for the actual roundtrip: rs #[test] fn test_roundtrip() { use serde_with::Same; let container = crate::macros::new_struct! { #[serde_as] #[derive(Serialize, Deserialize, PartialEq)] TestContainer { #[serde_as(as = "Vec<JsonString<Same>>")] pub values: Vec<Human> = fake::vec![Human; 50], } }; let serialized = serde_json::to_string(&container).unwrap(); let parsed = serde_json::from_str(&serialized).unwrap(); fn _assert_same_type<T>(_: &T, _: &T) {} _assert_same_type(&container, &parsed); for (expect, actual) in std::iter::zip(container.values, parsed.values) { assert_eq!(&expect, &actual); } } the TestContainer type cannot be named because it's from a macro (which is kind of weird, but I don't know the macro visibility rules exactly)

bronze fiber Jun 29, 2022, 5:11 PM

#

slow slate hey, I managed to get it working, I agree that implementing the `*As` traits it ...

How can T be coerced to ```rust
let wrapped: DeserializeAsWrap<T, TAs> = super::with::deserialize(deserializer)?;

#

I had Base62 as a wrapper for the case where something can be serialized into T: Into<u128>

bronze fiber Jun 29, 2022, 6:32 PM

#

And thank you Bruh!

slow slate Jun 29, 2022, 7:06 PM

#

bronze fiber How can `T` be coerced to ```rust let wrapped: DeserializeAsWrap<T, TAs> = super...

it's that you specify you want to deserialize a DeserializeAsWrap, it's a normal struct that implements Deserialize

bronze fiber Jun 29, 2022, 8:48 PM

#

slow slate it's that you specify you want to deserialize a `DeserializeAsWrap`, it's a norm...

What would that look like as an expression instead of an assignment?

slow slate Jun 29, 2022, 8:51 PM

#

you'd turbofish the T argument

bronze fiber Jun 30, 2022, 4:27 AM

#

Well I sincerely thank you for the assistance. I had gotten as far as Serialize but I had no idea how to nest with DeserializeAsWrap. I should have realized. How does a for bound work?

#

https://github.com/spikespaz/awaur/commit/3b893a43430cbb37a5985fdf08d658d5b727d0d4

GitHub

Use wrapper type for nesting inside JsonString · spikespaz/awaur@3b...

bronze fiber Jul 1, 2022, 6:18 AM

#

@slow slate Curious, I was browsing through the code of serde_with again and wondered why I don't need a marker field on my Base62 type?
https://docs.rs/serde_with/latest/src/serde_with/lib.rs.html#699

slow slate Jul 1, 2022, 11:07 AM

#

bronze fiber <@170999103482757120> Curious, I was browsing through the code of `serde_with` a...

I don't see how the example you linked is relevant

#

I assume that the Base62 will de/serialize always as the same type, without nesting

#Help implementing serde wrapper types for nested serialization behavior