Cannot resolve node packages even with "use node" | Convex Community | Page 1

atomic pine Dec 28, 2023, 4:28 AM

#

I have "use node" indicated in my actions file, and that is the only place in my entire code where I am using the Node APIs. However, I am getting these errors:

✘ [ERROR] Could not resolve "buffer"

    node_modules/safer-buffer/safer.js:5:21:
      5 │ var buffer = require('buffer')
        ╵                      ~~~~~~~~

  The package "buffer" wasn't found on the file system but is built into node. Are
  you trying to bundle for node? You can use "platform: 'node'" to do that, which
  will remove this error.

✘ [ERROR] Could not resolve "string_decoder"

    node_modules/iconv-lite/encodings/internal.js:49:28:
      49 │ var StringDecoder = require('string_decoder').StringDecoder;
         ╵                             ~~~~~~~~~~~~~~~~

  The package "string_decoder" wasn't found on the file system but is built into
  node. Are you trying to bundle for node? You can use "platform: 'node'" to do that,
  which will remove this error.

atomic pine Dec 28, 2023, 7:07 AM

#

Upon more testing, it seems like the actions are indeed using the node APIs buffer and string_decoder correctly (as part of iconv-lite). But it still throws these errors every single time I save any file which is distracting me from other genuine errors

atomic pine Dec 28, 2023, 9:27 AM

#

atomic pine Upon more testing, it seems like the actions are indeed using the node APIs buff...

Follow up: very bad news, this fails on convex deploy so I cannot push this to prod

wise flower Dec 28, 2023, 9:37 AM

#

Are you importing this "use node" file from any non-use node files?

#

a file with "use node" can use node modules for any actions it exports

atomic pine Dec 28, 2023, 9:40 AM

#

If I call the action from a non use node mutation does that count?

#

#

generateVocabCardsAction is from the use node file

wise flower Dec 28, 2023, 9:41 AM

#

That's fine, it's only importing that is a problem.

atomic pine Dec 28, 2023, 9:41 AM

#

hmm

wise flower Dec 28, 2023, 9:42 AM

#

and are this iconv-lite imported in any other files?

atomic pine Dec 28, 2023, 9:42 AM

#

only this file

#

it's not imported directly by this file but it is imported by another file that this file imports

wise flower Dec 28, 2023, 9:44 AM

#

"use node" causes that file to be bundled differently such that it can no longer declare mutations and queries and it can use node APIs like "string_decoder"

atomic pine Dec 28, 2023, 9:44 AM

#

yes

#

but that file has nothing except 2 actions both of which use node APIs

wise flower Dec 28, 2023, 9:45 AM

#

so any file that does not have "use node" will cause thus error if it imports that library

#

hm that sounds fine then

atomic pine Dec 28, 2023, 9:45 AM

#

yes

#

it's very strange but i am stuck now because i cant deploy

wise flower Dec 28, 2023, 9:47 AM

#

hm, I'd try removing files from the convex directly to see which one is causing this issue (assuming there's no indication of which file was being bundled when this error occurs)

atomic pine Dec 28, 2023, 9:47 AM

#

i know exactly which file

#

which i remove the one that uses iconv-lite the errors stop

wise flower Dec 28, 2023, 9:47 AM

#

One of the convex files that doesn't use "use node" is causing it

atomic pine Dec 28, 2023, 9:47 AM

#

wise flower One of the convex files that doesn't use "use node" is causing it

nope

#

it's this file specifically

wise flower Dec 28, 2023, 9:48 AM

#

or "use node" isn't working

atomic pine Dec 28, 2023, 9:48 AM

#

when the imported file stops importing iconv-lite the error immediately goes away

wise flower Dec 28, 2023, 9:48 AM

#

what does the top of that file look like?

#

the "use node" part

atomic pine Dec 28, 2023, 9:49 AM

#

#

"@/lib/functions/openai/vocab" uses iconv-lite (because openai has a bug now where they return non utf8 strings as utf8 so we need to manually decode)

wise flower Dec 28, 2023, 9:51 AM

#

huh could you double check that "vocab" isn't being imported from somewhere else?

atomic pine Dec 28, 2023, 9:51 AM

#

confirm

#

wise flower Dec 28, 2023, 9:53 AM

#

Not openai/vocab, this file generation/vocab

atomic pine Dec 28, 2023, 9:53 AM

#

wise flower Dec 28, 2023, 9:54 AM

#

it might be "./vocab", but looks like you always use the top level imports, nice

#

Would you be able to share this code? I'm [email protected], you could zip and email or share a GitHub repo with thomasballinger on GitHub

atomic pine Dec 28, 2023, 9:55 AM

#

Sure

#

Let me send it to you now

#

@wise flower added you on GitHub

wise flower Dec 28, 2023, 9:58 AM

#

cloning to take a look

atomic pine Dec 28, 2023, 10:00 AM

#

thank you

wise flower Dec 28, 2023, 10:13 AM

#

I think I see convex/server/meaning.ts --imports--> handle-duplicate-meanings.ts --imports--> "decode-openai" --imports--> iconv-lite

atomic pine Dec 28, 2023, 10:14 AM

#

ah i see

#

i only imported the zod schemas

#

but seems like it might have traced the other imports too

wise flower Dec 28, 2023, 10:15 AM

#

hm yeah it doesn't seem to be able to tree-shake this

atomic pine Dec 28, 2023, 10:15 AM

#

ok let me move the schemas to it's own file and try again

wise flower Dec 28, 2023, 10:15 AM

#

("it" == esbuild here the way it is invoked by convex)

atomic pine Dec 28, 2023, 10:17 AM

#

also one more problem

#

zid() resolves to a string for some reason

#

not a big problem because i can override it but slightly annoying

wise flower Dec 28, 2023, 10:20 AM

#

Huh interesting, could you file that in https://github.com/get-convex/convex-helpers/issues

atomic pine Dec 28, 2023, 10:21 AM

#

sure

#

#

wise flower Dec 28, 2023, 10:21 AM

#

thanks

#

ah I see, hm

atomic pine Dec 28, 2023, 10:21 AM

#

my zid is the same one i copied from the convex helpers file

wise flower Dec 28, 2023, 10:23 AM

#

oh so it's the same code, but not actually the one imported from the library, got it — curious if it repros in the code. Ian recently has been packaging some of these up to make them importable

atomic pine Dec 28, 2023, 10:23 AM

#

from convex/helpers right?

#

that package gives me super weird typescript errors when i install it

wise flower Dec 28, 2023, 10:24 AM

#

https://www.npmjs.com/package/convex-helpers

atomic pine Dec 28, 2023, 10:24 AM

#

yup this one, i tried installing it and it started throwing weird typescript errors

#

it has some magic where it is able to throw typescript errors from node_modules even though it is ignored according to tsconfig

wise flower Dec 28, 2023, 10:25 AM

#

yeah that's it, cool it's under very active development so would be good to get a repro of that sometime

atomic pine Dec 28, 2023, 10:25 AM

#

ok cool

#

will check it out again

wise flower Dec 28, 2023, 10:26 AM

#

I'm headed to bed, good luck + thanks for reporting these

atomic pine Dec 28, 2023, 10:27 AM

#

thank you very much for saving my repo

#

have a good rest!

#Cannot resolve node packages even with "use node"