#Issue with parsing data

1 messages · Page 1 of 1 (latest)

delicate wigeon
#

I'm pretty new to rust so this question might not necessarily belong here, but any help would be appreciated regardless.
I'm currently receiving some BSON data that I'd like to turn into a struct that I can modify, so currently I'm using twilight-model and serde_json for this.

let bson = bson::Bson::try_from(message.data.clone())?;
let data: twilight_model::gateway::payload::incoming::MessageCreate =
    serde_json::from_value(bson.into())?;

This gives me the following error: Error: missing field 'flags'
However, MessageCreate.flags is an Option, shouldn't that mean that even if it's missing, it should be able to parse the rest anyways? Is serde_json the right tool for this, or does twilight have it's own tools to serialize data like this. If not, what am I supposed to change to make this work?

tidal kraken
#

What gives you that error?

#

What is the value of bson after that try_from?

#

Also if there exist a serde_bson you should be able to use that directly

delicate wigeon
#

im running everything in docker so its a little harder to troubleshoot sadly

tidal kraken
#

Do you have a example the original data?

delicate wigeon
# 
BSON: Document({"attachments": Array([]), "author": Document({"accent_color": Null, "avatar": String("ab9021d467200645d446ef908797558d"), "banner": Null, "bot": Boolean(false), "discriminator": String("4444"), "id": String("659412484331143177"), "username": String("dissolved girl"), "public_flags": Int64(0)}), "channel_id": String("748027551338725408"), "content": String("Ur shite"), "edited_timestamp": Null, "embeds": Array([]), "flags": Int64(0), "guild_id": String("748024921807913060"), "id": String("1081311486468685936"), "type": Int32(0), "member": Document({"avatar": String("06f07ef7c53c97d9d3be05832425565f"), "communication_disabled_until": Null, "deaf": Boolean(false), "joined_at": String("2022-12-31T01:15:26.033000+00:00"), "mute": Boolean(false), "nick": String("mama"), "premium_since": String("2023-02-17T00:24:51.864000+00:00"), "roles": Array([String("1041382165268664390"), String("1043600065622712360"), String("1040764411788410921"), String("1081031724437344296"), String("1040367638439800942"), String("1040359982983557200"), String("1068619605230243891"), String("1040367515890618420"), String("1040369172313559060"), String("1069375908110155929"), String("947187020147593246"), String("1041093352994643999")]), "user": Null}), "mention_everyone": Boolean(false), "mention_roles": Array([]), "mentions": Array([]), "pinned": Boolean(false), "sticker_items": Array([]), "timestamp": String("2023-03-03T20:25:48.170000+00:00"), "tts": Boolean(false)})

This is the data after the try_from

#

sorry i thought you meant after the .into()

#

weird cause flags is definitely present there

delicate wigeon
#

changing it to

let data: twilight_model::gateway::payload::incoming::MessageCreate =
    bson::from_bson(bson)?;

which is probably the more logical thing to do, still gives me the same error actually

tidal kraken
#

Can you try to make the binary blob and send that, then I can see if I can figure out the issue

delicate wigeon
#

How would i do that?

tidal kraken
#

How do you make the events?

delicate wigeon
#

The events come in through redis streams as bson

#

message.data is RawBson, which is the first thing i can interact with here

tidal kraken
#

You should be able to write the raw bson to a file then?

delicate wigeon
#

oh yeah ill try that wait

#

sorry having a bit of trouble lol, working on it

tidal kraken
#

Also may I ask what gateway proxy thing that publishes it?

delicate wigeon
#

its spec-tacles/gateway

#

still trying to figure out how to turn the rawbson into something i can actually write into a file

tidal kraken
delicate wigeon
#

here you go :D

tidal kraken
delicate wigeon
#

I can try upgrading twilight in it if you think itll help

tidal kraken
#

give me 10 and I see if I can figure the issue out

delicate wigeon
#

sure thing, very much appreciated!

#

Seems to actually have to do with the versioning, downgrading twilight-model to 0.13.5 in my project (same version as in spectacles) actually fixes the issue

tidal kraken
#

Yeah it is because of the version, just found the issue

delicate wigeon
#

I assume that if i were to upgrade both sides to 0.15, itd work fine as well?

tidal kraken
#

flags have been added to the PartialMember object since 0.13 and that value should always be there

tidal kraken
delicate wigeon
#

ahh okay, im gonna try that then, always good to stay updated anyways

#

thank you very much for helping!

delicate wigeon
#

Another question, if its okay if i ask in the same thread. Can anyone give me an example of how to use DispatchEventWithTypeDeserializer?

native garnet
#

You probably want to use GatewayEventDeserializer instead which operates on

{
  "op": 0,
  "d": {},
  "s": 42,
  "t": "GATEWAY_EVENT_NAME"
}

DispatchEventWithTypeDeserializer is almost an implementation detail, it expects the "d" field's data as input + the "t" field's data as a seed.

#

And now that I think about it, you probably don't even want GatewayEventDeserializer either, twilight_gateway::parse calls it for you with the correct options.

delicate wigeon
#

Also GatewayEventDeserializer wont be possible for me as all i have is the event name string

native garnet
native garnet
delicate wigeon
#

Well I'm using spectacles as a gateway, which handles the gateway connection and then only sends the event name and dispatch data to redis

#

So pretty much i get completely untyped bson data into my application, what i wanna do now is use the event name string to parse every dispatch event into a typed struct

#

Of course i could just write all event names down manually but I'd prefer to use pattern matching so i was hoping there was an easy way to do the conversion

native garnet
#

Just use the event name to construct DispatchEventWithTypeDeserializer and call deserialize on a bson deserializer of the dispatch data

delicate wigeon
#

bsor? I got to the first part but i had no idea what to pass into the deserialize() function after i made the deserializer

#

Again I'm very new to rust haha so forgive me if I'm asking stupid questions

native garnet
#

Something which implements Deserializer, for instance we use serde_json::Deserializer (or simd_json`) when deserializing directly from Discord. I'm unfamiliar with bson, but I assume they also have a type which implements deserializer (they support serde?)

native garnet
delicate wigeon
#

Ohh you're referring to bson? Yeah i already convert the bson data into just a string for the event name

native garnet
#

How's the event data encoded?

delicate wigeon
#

The event data itself is also bson but i already managed to figure out how to convert that into a twilight struct

#

It's just that i didn't wanna write 50 if statements checking if the event name equals MESSAGE_CREATE and so forth

native garnet
delicate wigeon
#

Oh just tried that, realized the event name is actually of type Bytes

#

I can convert that into a string easily enough, (String::from_utf8_lossy(&message.event);, assuming thats the correct way)

#

This is as far as i got pretty much

let event_name = String::from_utf8_lossy(&message.event);
let deserializer = event::DispatchEventWithTypeDeserializer::new(&event_name);
let event = deserializer.deserialize(/* what goes here */)?;
native garnet
#

I'd just use String::from_utf8 as should the string not be UTF_8 it'll fail even if parsed lossy. You pass bson::Deserialize to the .deserialize method. You get bson::Bson (required for bson::Deserialize) by deserializing the "raw" data via https://docs.rs/bson/latest/bson/fn.from_slice.html (returns T: Deserialize, bson::Bson implements `Deserialize)

from_slice in bson - Rust

Deserialize an instance of type T from a slice of BSON bytes.

delicate wigeon
#

hmm i think the bson part is only relevant for the dispatch data itself, not for the event name

#

the event name is of type Bytes from the bytes crate

delicate wigeon
#

I keep having more questions btw, is it ok if i ask them all here or should i make different threads?
Either way for this one, is there an easy way to convert a vec of EventType to EventTypeFlags? EventType implements serialize/deserialize which is important for me cause i load in the event types through a config file, that doesnt seem to be the case for EventTypeFlags, but builder.event_types() doesnt support EventType, only EventTypeFlags and From<&EventType> is not implemented for EventTypeFlags

native garnet