#TypeScript API Generation Error [convex helpers]

69 messages · Page 1 of 1 (latest)

open beacon
#

Any ideas what i can do to troubleshoot this? convex helpers 0.1.63 - also happened with 0.1.62 convex 1.1.17

npx convex-helpers ts-api-spec
Failed to generate TypeScript API spec:  SyntaxError: Expected double-quoted property name in JSON at position 511616
    at JSON.parse (<anonymous>)
    at _Command.<anonymous> (/code/node_modules/.pnpm/convex-helpers@0.1.63_convex@1.17.0_@clerk+clerk-react@5.13.1_react-dom@18.3.1_react@18.3.1___ic44yhh7a2uwtef64brnfm7ef4/node_modules/convex-helpers/bin.cjs:3584:42)
    at _Command.listener [as _actionHandler] (/code/node_modules/.pnpm/convex-helpers@0.1.63_convex@1.17.0_@clerk+clerk-react@5.13.1_react-dom@18.3.1_react@18.3.1___ic44yhh7a2uwtef64brnfm7ef4/node_modules/convex-helpers/bin.cjs:1346:21)
    at /code/node_modules/.pnpm/convex-helpers@0.1.63_convex@1.17.0_@clerk+clerk-react@5.13.1_react-dom@18.3.1_react@18.3.1___ic44yhh7a2uwtef64brnfm7ef4/node_modules/convex-helpers/bin.cjs:2018:69
    at _Command._chainOrCall (/code/node_modules/.pnpm/convex-helpers@0.1.63_convex@1.17.0_@clerk+clerk-react@5.13.1_react-dom@18.3.1_react@18.3.1___ic44yhh7a2uwtef64brnfm7ef4/node_modules/convex-helpers/bin.cjs:1929:16)
    at _Command._parseCommand (/code/node_modules/.pnpm/convex-helpers@0.1.63_convex@1.17.0_@clerk+clerk-react@5.13.1_react-dom@18.3.1_react@18.3.1___ic44yhh7a2uwtef64brnfm7ef4/node_modules/convex-helpers/bin.cjs:2018:31)
    at /code/node_modules/.pnpm/convex-helpers@0.1.63_convex@1.17.0_@clerk+clerk-react@5.13.1_react-dom@18.3.1_react@18.3.1___ic44yhh7a2uwtef64brnfm7ef4/node_modules/convex-helpers/bin.cjs:1848:31
    at _Command._chainOrCall (/code/node_modules/.pnpm/convex-helpers@0.1.63_convex@1.17.0_@clerk+clerk-react@5.13.1_react-
# 
dom@18.3.1_react@18.3.1___ic44yhh7a2uwtef64brnfm7ef4/node_modules/convex-helpers/bin.cjs:1929:16)
    at _Command._dispatchSubcommand (/code/node_modules/.pnpm/convex-helpers@0.1.63_convex@1.17.0_@clerk+clerk-react@5.13.1_react-dom@18.3.1_react@18.3.1___ic44yhh7a2uwtef64brnfm7ef4/node_modules/convex-helpers/bin.cjs:1844:27)
    at _Command._parseCommand (/code/node_modules/.pnpm/convex-helpers@0.1.63_convex@1.17.0_@clerk+clerk-react@5.13.1_react-dom@18.3.1_react@18.3.1___ic44yhh7a2uwtef64brnfm7ef4/node_modules/convex-helpers/bin.cjs:1989:23)
#

Same thing happens with open-api-spec, different line.

jagged ledge
#

Hi! Can you please try running npx convex function-spec then running npx convex-helpers ts-api-spec --input-file [file name] on the output? I want to see if you get the same error

open beacon
#

the output is 8mb and the file is 222k. If that has anything to do with hit.

jagged ledge
#

Which output is 8mb and which file is 222k? I will look into this issue further this week. Let me know if you run into any issues with the workaround

open beacon
#

input to convex helpers is 8mb and it generates 222k

open beacon
#

To me its a little weird that it adds .ts automatically when you specify --output-file with convex-helpers

#

I'm thinking it would be nice to script up something to have this run every time at the start of when a bundle is sent. Are both commands entirely local? its taking less than 7 seconds to run both commands so it could be done in the time it takes to bundle. But i think triggering it would be the tricky part?

jagged ledge
#

It adds .ts because you are generating a TypeScript file with this command. The second command is local, but the first one asks your Convex deployment for the function args/returns validators that are currently deployed.

jagged ledge
#

I will work on the command this week! In the meantime, let me know if you run into any more issues

jagged ledge
#

I identified the problem and am working on a fix. I will have an update for you next week. Thanks for your patience!

jagged ledge
#

can you please try out this version of convex-helpers and tell me if it resolves your issues 0.1.64-alpha.1?

open beacon
#

Also, I was curious if you had any ideas about how to run this automatically after each bundle runs?

jagged ledge
#

awesome! i'm not sure. would you like to run this after every npx convex deploy or npx convex dev? what is your specific use case?

open beacon
steady wadi
#

Depending on how you have your monorepo set up, you could add this to your build step in your package.json

open beacon
open beacon
# jagged ledge can you please try out this version of convex-helpers and tell me if it resolves...

Sorry, I didn't do a full test, upon further testing i do believe its broken, its now only generating 382 lines vs 7500~ running the short command or manual. I get a seemly random selection of the api.

npx convex-helpers ts-api-spec --output-file ../shared/src/lib/api
npx convex function-spec > tempfile; npx convex-helpers ts-api-spec --input-file tempfile --output-file ../shared/src/lib/api; rm tempfile
#

But the syntax does pass

jagged ledge
#

is it just excluding your internal functions? or is it totally random?

open beacon
wind rampart
#

Seeing this as well, it does not seem to reliably generate the spec file for all of our APIs

jagged ledge
open beacon
jagged ledge
open beacon
#

it might be handy to have -v for version

jagged ledge
#

i think you can also run ‘npm list convex-helpers’ to see which version you are on.

open beacon
#

oh yeah its back to 0.1.63.. i have less trust with mono-repo that way.

jagged ledge
#

okay, it looks like you are getting the same problem on both versions. i’m going to do a bit more digging. sorry about these issues.

#

is the result from ‘npx convex function-spec’ also shorter than before?

open beacon
#

yeah the output is 1.8mb/48,000 lines and i think it was 8mb/200k~ before,

woeful elbow
#

@open beacon what is your instance name? @jagged ledge is out this week so I'll be continuing the investigation

open beacon
woeful elbow
# open beacon confident-gerbil-241

Do I have permission to view your source code? (This is to confirm that metadata about your functions is written correctly to the db). If you have a function name that is missing from your generated spec, that would also be helpful.

#

Does running npx convex function-spec deterministically generate the same output if you run it a few times?

open beacon
open beacon
open beacon
#

Yeah the dashboard thinks i only have 107 when i should have 400+

woeful elbow
#

yep, we've found the issue and are rolling out the fix. your deployment should still be running and your app should still be working, but some functions might not appear in the dashboard and function-spec will be missing some functions. Your functions still exist and should still work if you run them.

open beacon
#

Ok yeah, i just got around that on production with cli run

woeful elbow
#

@open beacon can you try generating openapi spec?

open beacon
#

npx convex function-spec - its 6.4mb at 177k lines so a lot bigger now

woeful elbow
#

Are there functions still missing?

open beacon
#

npx convex-helpers ts-api-spec is still broken unless i manually feed it the function spec file (i think that is expected)

#

im 99% sure all the functions are being generated

#

at least public ones

woeful elbow
#

great! Could you try ts-api-spec again after upgrading to convex-helpers to 0.1.64-alpha.1?

open beacon
carmine river
#

I'm guessing your aim is to use the generated spec rather than importing from your backend package because your massive convex-ents app has brought typescript to its knees? (I went down a similar path, but removing zod validators + deriving types from return validators helped a lot)

You can't just pipe the output of convex dev or convex logs (why?) into a file easily - I hoped to use the success message to trigger a generation. Maybe there's some way to read the text from the console, or capture it with some wrapper (turborepo, vscode?)

The other way I thought of is to write your own file watcher that triggers convex dev --once - that returns success/fail exit codes so it's only triggered on a successful push

#

Just writing out your types might be easier than all this 🙂

steady wadi
carmine river
#

Ah right - the actual issue with logs is that it doesn't report the "(user) updated the configuration" message that the dashboard does, or at least I couldn't find a way to see that

steady wadi
#

ok useful to hear you'd like to be able to generate these, this could even be a standard dev flow like `npx convex dev --generate-ts-api api.ts

#

its important this work in a monorepo, so far the suggestion has been to export that _generatex/api file but generating APIs was always in the plan, since it's required for Python, Rust, and other language than TypeScript

carmine river
#

Yeah, I imagine this isn't the exactly the imagined primary use case, especially when your actual convex code is in the folder next door.

Again, I'm just guessing that Typescript woes are the reason ampp wants to do this, because his story sounds similar to mine.

I know all big typescript projects suffer from this in some degree - but this feels like a pretty heavy fix for the problem. I don't know if there's much else you can do than write your own return types etc.

It's sad watching your incredible Typescript DX slowly fade away as your awesome convex app becomes huge. It's kinda the double-edged sword of your fancy React button being so "literally" connected to your table schema

steady wadi
#

To the degree it's inevitable we can make that transition smooth by added compilation steps, the structure of api was chosen so that you could build these objects manually. But there are some approaches to keep it further, things like what TanStack Router is doing with their route types.

#

@carmine river when you wanted this, where were you generated the api object? In the repo using it, or in the repo where the convex folder is? If in the repo where the API is defined, how were you importing it in the repo where it was used?

carmine river
# steady wadi <@184670179698016256> when you wanted this, where were you generated the api obj...

My plan was to generate them in the backend package (that contains only my convex code), using the output file option to drop the generated API into the package that required use it, eg. packages/backend -> packages/web. Nothing would be imported from backend - as if it were in an unrelated repo. This way I don't have to wait for typescript to recheck the entire backend every few seconds when it isn't changing (incremental: true doesn't solve this), because the types are baked in.

"generate-api": "mkdir -p out && convex function-spec > out/fn-spec.json && convex-helpers ts-api-spec --input-file out/fn-spec.json --output-file ${OUTPUT_FILE:-out/convex-api}",
"generate-api-fixed": "convex-helpers ts-api-spec --output-file ${OUTPUT_FILE:-out/convex-api}"

Since it is all still typescript, I was wondering if there was some sort of declaration file based solution for this, since we already literally have this information locally. Maybe it'd be better than hitting your backend code generator on every push. I don't know if this is possible/worth it

steady wadi
#

The mid-term solution here is probably local backend, or a convex-test -like solution of building the API of these functions (still using output validators) by importing the modules and walking over the imports.

open beacon
#

As far as typescript generation i got the fastest single core performance in native linux so its been fairly good as typescipt only hits 1 core hard on the 9950x. But im thinking of trying biome soon.

steady wadi
#

cause apps/web1 apps/web2 to try to build the convex folder
and this build fails? or takes longer?

open beacon
#

it fails with 100s of errors