RSA with openssl vs RustCrypto/RSA

[torn egret]

so your first mistake is using RSA 👉😎👉

do you have a minimal reproducible example?

You say verifying singatures doesn't work but your code doesn't show how these signatures are generated or how they are verified.

[sharp crescent]

yeah, the signature generation/verification code are the likely culprit

the rsa crate API is definitely not great

currently the most straightforward thing you can do is almost always wrong

(specifically in regard to PKCS#1 v1.5 signatures)

[gleaming spear]

Generating / Verifying signatures with the RSA crate does work, but only when it loads a privatekey that was made with the OpenSSL crate

Is there a better crate I should try if RSA is a problem?

I really want to avoid an OpenSSL dependency as it was a hassle in the previous versions

[gleaming spear]

rsa implementation: https://github.com/BrettMayson/HEMTT/tree/main/libs/signing

openssl implementation: https://github.com/KoffeinFlummi/armake2/blob/master/src/sign.rs

[sharp crescent]

are you handrolling PKCS#1 v1.5? or is that something different

[gleaming spear]

honestly no clue what the formats here are, I'm just using the existing methods from the openssl implementation. I am not overly familiar with the inner workings of RSA. new https://github.com/BrettMayson/HEMTT/blob/main/libs/signing/src/signature.rs#L23-L45 vs old https://github.com/BrettMayson/HEMTT/blob/main/libs/signing/src/signature.rs#L23-L45

The formats are custom for the video game Arma 3, with very little documentation (and none of it official) on how they work. Just reverse engineering and code sharing within the community.

[sharp crescent]

aah

[gleaming spear]

I'm just unsure of what could be wrong with the private key generation, like I've said, new or old works with a private key made by the old code, both new and old will generate signatures with a new private key, but they are both invalid

Both files look the same, same length of every value. But it's just a bunch of numbers, no clue where it is going wrong and not sure how to debug.

[gleaming spear]

Issue has been found and resolved, thanks for taking the time to help