decryption problem

[dusk glade]

AEAD is fundamentally not a stream construct

it must be opened in full, with the entire ciphertext and tag.

in your Read call inside (*StreamDecryptor).Read, you're not guaranteed to get len(p)+sd.gcm.Overhead() bytes. you're only guaranteed to get up to that many bytes.

similar thing in (*StreamEncryptor).Read. the underlying reader could return a single byte for every Read call, so you'd be producing an entire ciphertext and tag pair for every single byte.

if you're ever depending on Read reading a certain number of bytes, you're doing something wrong. (ok, there are a few readers that guarantee they fill the buffer, but unless you're limiting yourself to those, you shouldn't make that assumption.)

[north canopy]

Hi Carson, Thank you for the quality answer. If my primary goal is performance should I use some other symmetric encryption? Or is the whole notion of "stream encrypting/decrypting" on the fly misguided? Is reading the whole request/response to memory and then encrypting/decrypting really the most optimal solution?

[dusk glade]

it is using the AEAD interface. there's no way around buffering the entire plaintext/ciphertext.

GCM can be used in a stream fashion, but Go's standard library doesn't provide you the means to do so. it's a bit of a footgun-prevention mechanism.

you start to open yourself up to, e.g., oracle attacks if you're not careful.

(the main concern is that you get access to decrypted but unauthenticated data. the AEAD interface provides no such way of doing so, so there's no possibility of things going wrong.)

that's fundamentally the case with any stream-based encryption, as the very point of a stream is that you don't have to buffer data while waiting for the tag (which is the only way that authentication happens).

so, think: what would happen with the data you already processed if authentication fails at the end? you can't tell me you didn't process it, because that means you buffered it! 😛

[north canopy]

Ah I see. Thanks for the explanation. I should probably look into ChaCha20 or something then.

[dusk glade]

i'm not sure what difference you're looking for.

what i said applies fundamentally to all encryption, not GCM.

ChaCha20-Poly1305 (what i'm guessing you probably meant by ChaCha20) is the same.

[north canopy]

If GCM can be used in a stream fashion, but is not enabled by the go std lib like discussed, I thought there surely would be a "streaming focused" solution. I googled "streaming chiper" and ChaCha20-Poly1305 was the first thing that popped up

[dusk glade]

ok, but you basically ignored what i asked here then 😁

yes, you can use a stream cipher in a stream fashion, but not in an authenticated manner

[north canopy]

There must be some way to not load huge files into memory to encrypt them.

[north canopy]

Oh yes. Sorry I am a total beginner with cryptography. Thanks for the patience.

[dusk glade]

i understand. i just want to get across the fundamental idea: streaming means that you don't keep everything in memory, i.e. you can't undo what has already been processed. authentication only happens once. those two ideas are fundamentally at odds.

you could consider breaking your data into discrete chunks to AEAD individually, but then you're no longer authenticating the order or existence of any particular data. that can be overcome by using, e.g., a sequence number for each chunk to prevent reordering and confirm that everything is present.

that's the strategy used by, e.g., TLS, the protocol that we're using to talk to each other (or at least talk to Discord on each of our ends) right now.

[undone estuary]

@north canopy

https://github.com/SimpaiX-net/simpa/blob/main/engine/crypt/crypter.go

see a working robust example

2 examples of robust aes ctr hmac and aes gcm hmac

[dusk glade]

i'm not sure you understand what they're going for, those both require the full ciphertext/plaintext up front

that's what they already have, and the AEAD interface provides a more standardized and safe way of doing that

(and you know GCM already provides authentication, right…? there's no point in adding on an HMAC for GCM)

[north canopy]

I am looking into implementing this sequence numbering approach, would it make more sense to use a pre-determined fixed chunk size or some kind of delimiter/marker? Which is more common?

[dusk glade]

i'd say that unless your data is deeply, intrinsically a fixed size, having e.g. a length prefix gives you a lot more flexibility

again, there are multiple length fields in every fragment sent in your connection to discord. does it add a little overhead? sure. but everyone survives

[north canopy]

I went with layout "nonce (24 B) | payload (n - 4096 B) + overhead 16 B | mac of seqNums 64B (only last chunk)". I now have working encryption and decryption on the fly in memory. There is one problem tough. The seqNum is authenticated in AEAD on each chunk decryption so order of the chunks is safe. But if a chunk is removed the attacker could possibly re-use a MAC from another file if they are known to be the same length (have as many chunks)

[dusk glade]

a "sequence number" is typically really a nonce in the high bits plus a counter in the low bits

as long as you pick a long enough nonce randomly, it won't be an issue

i'm not sure what the mac of sequence number construction you're going for is there. the validity of the counter can be implicit when used as part of the nonce.

[north canopy]

Here is the decrypt and encrypt for context : https://pastes.io/wkjwfmwvi5

[dusk glade]

something to note: nonces need not be independent from each other. it's perfectly valid for one nonce to differ from the last by only one bit. it's just that they can't be reused in their entirety. it's typical to choose one IV at the start of a "connection" (if you're planning on never separating these chunks) and simply add in or XOR in a sequence number so the nonce is guaranteed to be different (and that also implicitly protects the ordering of the chunks).

also: you're still in the same Read trap i mentioned yesterday. that Read call could read anywhere from zero to len(buffer) bytes. yes, you're using n which is good, but you're still going to fail decryption on a short read.

you might want io.ReadFull there (like you used earlier).

[north canopy]

i imagine the 24byte nonce is ok? should i use it to make a "sequence number", i am not sure

and thank you a lot for all the answers. you have been incredibly helpful.

[dusk glade]

24 bytes is certainly enough, but my point is you don't need to randomly choose one for every chunk and send it every time.

if you're always planning on sending the same chunks in the same order, you can generate an IV once, send it once, then use it as a basis for the nonce for each decryption. you just need to make sure that it changes from one chunk to the next, which is where something like XORing in a sequence number can come in.

like i said, nonces need not be random with respect to each other. they should probably be random with respect to any other sequence of encryptions (which is where the initial randomness of the IV comes in), but within a sequence of encryptions, you only need to make sure they're different, i.e. not all of the 192 bits match. 191 matching is totally fine.

[dusk glade]

QUIC (the underlying protocol of HTTP/3) uses precisely this strategy.

(it doesn't actually even send the IV, it derives it from the key exchange, but that's beside the point)

[north canopy]

Is doing something like this secure:
nonce := make([]byte, nonceSize) for i := range nonce { nonce[i] = initialNonce[i] ^ seqNumBytes[i] }

[dusk glade]

if len(seqNumBytes) == len(initialNonce) == nonceSize, sure.

XOR is a bijection, so x ^ a != x ^ b is always true for any a != b.

[north canopy]

Is doing it like this "XOR'ing" the optimal solution or is there a "better" alternative? Also should i change var seqNum uint32 = 0 to something else, is the predictability a problem? ( you mentioned : a "sequence number" is typically really a nonce in the high bits plus a counter in the low bits)

[dusk glade]

ok, i admit i was slightly loose with mixing my terminology there.

really should say: a nonce is often really an IV in the high bits plus a counter in the low bits. it's often loosely called a sequence number, just with a different starting place (which is what the IV gives you).

but the XOR works fine, too; you don't need to separate off the IV from the counter.

they can all mix in just fine. as long as you keep the property that you never reuse a sequence number, the nonce will never be the same.

[north canopy]

i assume "never reusing a sequence number" pertains to a single list of chunks. as it is guaranteed that every files encryption starts with the first chunk having seqNum=0 etc.

[dusk glade]

yeah, that much is fine, as long as the IV is random (so the nonce will never coincide between streams)

you can equivalently think of it as picking a starting sequence number at random and incrementing from there.

they're not really different.

just different ways of modeling it to end up at the same result.

(xor and addition aren't exactly equivalent, but they have similar properties when used in this way, and it's usually done like this to obviate the need for a big-integer library, since no platform has 24-byte integers natively)

[north canopy]

so i should probably remove the mac? i do not understand how the nonce guarantees that all chunks are there? i understand it guarantees the chunks that are present are in the right order.

[dusk glade]

that's a good catch! i hadn't fully considered that. with an interactive protocol, an attacker can always just choose to drop all packets from any point forward, so there's not much point in worrying about the "everything is present" case.

if you want to keep the plaintext "pure", i.e. only the bytes of the stream, you could have part of the chunk framing (i.e. outside the encryption) indicate whether it was the last, then have that be part of the associated data. it won't be encrypted, but it will be authenticated.

or you could just stick it in the plaintext.

or you could have some metadata that lists the number of chunks. different options.

[north canopy]

that's what the mac was intended for. the mac is calculated from the seqNumBytes, so the mac would not match if something was missing. the problem i see with that is that you can simply copy a "wrong" valid mac from another encrypted files end to make a file seem valid (remove one chunk, attach mac of one chunk shorter archive). this is all as i want that the decryptor can know that the file is "how it was intended", previously i have always just signed the whole encrypted file but that's not an option as it would require me to buffer the whole stream to memory and i am trying to support multi terabyte uploads (max 5TiB)

i initially thought about putting all the plaintext trough the mac but that is quite resource intensive

[dusk glade]

yeah, if your threat model is an untrusted server so external metadata isn't an option (and you don't know the number of chunks at the start), i'd probably just differentiate the last chunk.

[north canopy]

of course an attacker should never have access to modify the files in storage but good security comes trough paranoia right

[dusk glade]

all it takes is a single bit

[north canopy]

how can i do that since the stored chunks are not fixed width? i do not understand what you mean

[dusk glade]

how are you deciding how long a chunk is when reading? a length prefix?

or do you store them discretely?

[north canopy]

chunks are appended/written to a single file for each file. chunking is just for the encryption. decrypting is done by tryiing to read the stream in 4096 byte chunks (full) and then reading only 4096-n bytes if the chunk is only partial

[dusk glade]

okay. so the “associated data” component of AEADs allows you to authenticate but not encrypt data. that said, that data doesn’t actually need to exist next to the ciphertext. it can be implicit. e.g. when you discover the last chunk while encrypting you add some specific associated data, and when you discover it while decrypting you do the same.

[north canopy]

ah i see, thanks

[dusk glade]

if you try to decrypt what you think is the last chunk but actually isn’t because an attack occurred, it’ll fail opening.

[north canopy]

yeah, same way as with the seqnum that keeps the order in additional data

[dusk glade]

yeah! nonces and AD are similar in that they’re often implicit, and changing either will prevent decryption. they don’t have the same internal use or security properties otherwise, though.

i’ve gotta go, not because i’m not enjoying helping but i’ve just got an early morning ahead of me. 🙂

[north canopy]

yeah no problem at all. huge thanks for this!

[dusk glade]

yep! and i’ve gotta say, for someone new to the crypto sphere, you’ve got a really great understanding of the mindset needed for it. that chunk dropping didn’t even register to me

[north canopy]

thanks! means a lot.

[dusk glade]

i wouldn’t have expected most of those concepts to have clicked on the first time around, either.