MD5 a byteArray

[vital geyser]

I'm trying something which should be pretty trivial, setting up a basic hashing function that takes in a byteArray and spits out a hash data consistency purposes. Right now I'm using a internal Action that takes the ByteArray and uses the Node Crypto toolkit to digest a hash. This works well on smaller byteArrays but around a couple of MB in it starts failing with the following error:

"code": "[Request ID: 8cd403c86459d299] Server Error: Uncaught Error: Your request couldn't be completed. Try again later.",
    "trace": "Uncaught Error: Your request couldn't be completed. Try again later.\n    at async <anonymous> (../convex/http.ts:29:14)\n    at async invokeFunction (../../node_modules/convex/src/server/impl/registration_impl.ts:79:11)\n    at async invokeHttpAction (../../node_modules/convex/src/server/impl/registration_impl.ts:422:0)\n    at async <anonymous> (../../node_modules/convex/src/server/router.ts:322:16)\n"

Here is my function. What is the advices approach? Can somebody tell me recommended route?

export const getMD5Hash = internalAction({
  args: {
    buffer: v.bytes(),
  },
  handler: async (_, { buffer }) => {
    try {
      const hash = crypto.createHash("md5");
      const hashBuffer = Buffer.from(buffer);
      hash.update(hashBuffer);
      return hash.digest("hex");
    } catch (error) {
      console.error(error);
      throw new Error("Failed to get MD5 hash");
    }
  },
});

[high gorge]

Thanks for posting in #1088161997662724167.
Reminder: If you have a Convex Pro account, use the Convex Dashboard to file support tickets.

    - Provide context: What are you trying to achieve, what is the end-user interaction, what are you seeing? (full error message, command output, etc.)
    - Use [search.convex.dev](https://search.convex.dev) to search Docs, Stack, and Discord all at once.
    - Additionally, you can post your questions in the Convex Community's #1228095053885476985 channel to receive a response from AI.
    - Avoid tagging staff unless specifically instructed.

    Thank you!

[rancid prairie]

Hi @vital geyser, which of your projects is this on? This like a bug, there should either be a better error here about function argument sizes being limited to 8MB (sounds like you're below this) or this should work.

OK I see this on our side, will let you know when we know more.

[rancid prairie]

@vital geyser This is hitting an AWS Lambda size limit, larger than your actual bytearray size due to the way we encode bytearray arguments. We'll get back to you about improving this error or fixing, but the workaround I would suggest is not using "use node" for this internalAction. You may get better performance this way anyway, depending on how your internalAction is being called. You could also run this code in a mutation or query this way.

Instead of using Node.js APIs you'll want to use Web Crypto APIs.

This is what I got when I asked an LLM to translate your Node.js code to browser code, dunno if this is right — but generally if it runs in a browser it should run in the Convex JS runtime (instead of "use node" which is AWS Lambda).

const hashBuffer = await crypto.subtle.digest('MD5', arrayBuffer);
const hashArray = Array.from(new Uint8Array(hashBuffer));
const hashHex = hashArray
  .map(byte => byte.toString(16).padStart(2, '0'))
  .join('');

[unkempt garden]

You can also read larger data into actions by writing it to file storage, then passing the storage id to the function to read / stream

[vital geyser]

Yeah the problem sort of arises from the fact that the native crypto imp. does not support MD5. Reading the file from the file storage seems like a handy escape hatch indeed. Any recommended external package?

[vital geyser]

I eventually solved the issue with the use of Crypto-JS https://www.npmjs.com/package/crypto-js