#migrating from serde_json to serde_cbor for encoding messages

38 messages · Page 1 of 1 (latest)

stuck drum
#

Currently a rust project uses serde_json for encoding protocol messages which is useful for logging but I need to use serde_cbor for encoding messages (while logging will still use serde_json). I am not sure how do I proceed with this.
So far I have included serde_cbor in Cargo.toml and I am doing it like this:
Old:

line = line.trim_end().to_string();
                let message: TakerToMakerMessage = serde_json::from_str(&line).unwrap();
                log::info!("[{}] <=== {} ", maker_clone.config.port, message);

Now:

line = line.trim_end().to_string();
                let message: TakerToMakerMessage = serde_cbor::from_slice(line.as_bytes()).unwrap();
                log::info!("[{}] <=== {} ", maker_clone.config.port, message);

Old:

let message: MakerToTakerMessage = serde_json::from_str(&line)?;
    log::debug!("<== {:#?}", message);
    Ok(message)

Now:

let message: MakerToTakerMessage = serde_cbor::from_slice(line.trim().as_bytes())?;
    log::debug!("<== {:#?}", message);
    Ok(message)

Am I migrating it right? Because some tests are now failing (if someone needs some context: here is my commit)
In the logs, I have many

Custom { kind: InvalidData, error: "stream did not contain valid UTF-8" }

Thanks a lot and happy new year!

hasty hill
#

Um cbor is a binary format

#

So you shouldn't be using String with it

stuck drum
#

@hasty hill yes I understood it the hard way and later changed it to use Vec<u8>. Still no help so far

hasty hill
stuck drum
# hasty hill What error do you get now?

I am afraid it will require a bit context for you to understand
But on running the tests I get

...
[2024-01-08T19:23:55Z ERROR coinswap::maker] Error deserializing message: ErrorImpl { code: Message("invalid type: integer `0`, expected internally tagged enum TakerToMakerMessage"), offset: 0 }
[2024-01-08T19:23:55Z ERROR coinswap::maker] Error deserializing message: ErrorImpl { code: Message("invalid type: integer `0`, expected internally tagged enum TakerToMakerMessage"), offset: 0 }
...

Which resulted from this line:

let message: TakerToMakerMessage = match serde_cbor::from_slice(&buf) {
                    Ok(message) => message,
                    Err(e) => {
                        log::error!("Error deserializing message: {:?}", e);
                        continue;
                    }
                };
hasty hill
stuck drum
#

The enum has some custom data structures

#[derive(Debug, Serialize, Deserialize)]
#[serde(tag = "type", rename_all = "lowercase")]
pub enum TakerToMakerMessage {
    TakerHello(TakerHello),
    ReqGiveOffer(GiveOffer),
    ReqContractSigsForSender(ReqContractSigsForSender),
    RespProofOfFunding(ProofOfFunding),
    RespContractSigsForRecvrAndSender(ContractSigsForRecvrAndSender),
    ReqContractSigsForRecvr(ReqContractSigsForRecvr),
    RespHashPreimage(HashPreimage),
    RespPrivKeyHandover(PrivKeyHandover),
}
hasty hill
#

Yeah the error means the cbor it got was just the number 0

stuck drum
hasty hill
stuck drum
hasty hill
#

Sure

#

I'm expecting a 00 byte at the beginning

stuck drum
# 
log::info!("buf = {:?}", buf);

I get:

[2024-01-08T19:31:27Z INFO  coinswap::maker] buf = [0, 0, 0, 61, 163, 100, 116, 121, 112, 101, 106, 116, 97, 107, 101, 114, 104, 101, 108, 108, 111, 116, 112, 114, 111, 116, 111, 99, 111, 108, 95, 118, 101, 114, 115, 105, 111, 110, 95, 109, 105, 110, 0, 116, 112, 114, 111, 116, 111, 99, 111, 108, 95, 118, 101, 114, 115, 105, 111, 110, 95, 109, 97, 120, 0, 0, 0, 0, 19, 161, 100, 116, 121, 112, 101, 108, 114, 101, 113, 103, 105, 118, 101, 111, 102, 102, 101, 114]
hasty hill
stuck drum
#

I dont think so 😦

loop {
                let mut buf = Vec::new();
                select! {
                    read_ret = reader.read_to_end(&mut buf) => {
                        match read_ret {
                            Ok(0) => {
                                log::info!("[{}] Connection closed by peer", maker_clone.config.port);
                                break;
                            }
                            Ok(_) => (),
                            Err(e) => {
                                log::error!("error reading from socket: {:?}", e);
                                break;
                            }
                        }
                    },
                    _ = sleep(Duration::from_secs(maker_clone.config.idle_connection_timeout)) => {
                        log::info!("[{}] Idle connection closed", addr.port());
                        break;
                    },
                };
hasty hill
#

Aka where reader is getting it's data

stuck drum
hasty hill
#

That connects to this

stuck drum
#

I mean my exact changes are this branch that corresponds to the logs

hasty hill
stuck drum
#

did you generate this from the buf data? If yes, how?

hasty hill
#

After making it into hex

stuck drum
#

I see, yes then the buf looks right, isnt it?
What could be wrong then

hasty hill
#

Cbor works on one message at a time without a length prefix

stuck drum
#

so this migration isn't possible with serde_cbor? Should I look into other serde alternatives?
(sorry if i sound stupid, i am stupid in rust)

hasty hill
#

No, you just need to parse the messages correctly

stuck drum
#

ok thanks a lot, can u give some hint?
(Only if you are free)

stuck drum
#

ok i think ive made some progress

stuck drum
stuck drum
stuck drum
#

should write here, if im doing something wrong pls lmk!

if !buf.is_empty() {
                    // Read the length prefix (assuming it's a u32)
                    let len = u32::from_be_bytes(buf[0..4].try_into().unwrap()) as usize;

                    // Ensure the buffer has enough data
                    if buf.len() < len + 4 {
                        break;
                    }

                    // Get the message data
                    let message_data = &buf[4..4 + len];

                    // Deserialize the message
                    let message: TakerToMakerMessage = match serde_cbor::from_slice(message_data) {
                        Ok(message) => message,
                        Err(e) => {
                            log::error!("Error deserializing message: {:?}", e);
                            continue;
                        }
                    };

                    // Log the message
                    log::info!("[{}] <=== {} ", maker_clone.config.port, message);
                    log::info!("Message deserialised successfuly: {:?}", message);

                    // Remove the processed data from the buffer
                    // buf.drain(0..4 + len);

                    let message_result: Result<Option<MakerToTakerMessage>, MakerError> =
                        handle_message(&maker_clone, &mut connection_state, message, addr.ip())
                            .await;