How to encrypt and decrypt files from S3 without loading the hole content into memory?
I have this basic example (example in comments), I wonder is some of this can be used.
#How to encrypt and decrypt files from S3 without loading the hole content into memory?
pseudo cloud
// Encrypts data using AES-GCM
async function encryptData(data: string, key: CryptoKey) {
const encoder = new TextEncoder();
const iv = crypto.getRandomValues(new Uint8Array(12)); // Initialization Vector
const encrypted = await crypto.subtle.encrypt(
{
name: "AES-GCM",
iv: iv,
},
key,
encoder.encode(data),
);
return { encrypted, iv };
}
// Decrypts data using AES-GCM
async function decryptData(
encrypted: ArrayBuffer,
iv: Uint8Array,
key: CryptoKey,
) {
const decrypted = await crypto.subtle.decrypt(
{
name: "AES-GCM",
iv: iv,
},
key,
encrypted,
);
const decoder = new TextDecoder();
return decoder.decode(decrypted);
}
async function generateKeyString(): Promise<string> {
try {
const key = await crypto.subtle.generateKey(
{
name: "AES-GCM",
length: 256,
},
true,
["encrypt", "decrypt"],
);
const exportedKey = await crypto.subtle.exportKey("jwk", key);
return JSON.stringify(exportedKey);
} catch (error) {
console.error("Error generating key:", error);
return "";
}
}
async function generateKeyFromString(
keyString: string,
): Promise<CryptoKey> {
const keyData = JSON.parse(keyString);
const key = await crypto.subtle.importKey(
"jwk",
keyData,
{ name: "AES-GCM", length: 256 },
true,
["encrypt", "decrypt"],
);
return key;
}
// Example usage
async function example() {
const keyString = await generateKeyString();
const cryptoKey = await generateKeyFromString(keyString)!;
const stringToEncrypt = "Hello, world!";
const {
// Save the encrypted data and the IV to DB
encrypted,
iv,
} = await encryptData(stringToEncrypt, cryptoKey);
const decryptedMessage = await decryptData(encrypted, iv, cryptoKey);
console.log("Original:", stringToEncrypt);
console.log("Encrypted:", encrypted);
console.log("Decrypted:", decryptedMessage);
}
example();
rugged stoneBOT
I wasn't able to find the answer to your question in the official Deno documentation. However, based on your message, I can provide some general advice.
When dealing with large files and trying to avoid loading the entire content into memory, you would typically use streams. Streams allow you to process data incrementally as it is being read or written, which is ideal for handling large files.
In Deno, you can use the reader and writer interfaces along with the copy function from the std/streams module to work with streams. If you have an example in the comments, please ensure that it uses streaming methods to read from and write to S3.
Please note that this advice is not based on the Deno documentation, and I would advise you to double-check the accuracy of this advice. If you're using a third-party library for S3 interactions, make sure to consult the library's documentation for streaming support.