Importing a snapshot during deploy stored on s3 | Convex Community | Page 1

lavish comet Sep 25, 2024, 4:08 PM

#

Hi, my snapshot is now above 100mb. This means I can no longer commit it to github. Is there a way to tell the deploy command npx convex import --preview-name "$VERCEL_GIT_COMMIT_REF" ./data/mock-data/snapshot.zip during a deploy to tell it to grab the file from a remote resource, like an s3 bucket? What are workarounds for large snapshots that can't be commited to a repo?

stone boneBOT Sep 25, 2024, 4:08 PM

#

Thanks for posting in #1088161997662724167.
Reminder: If you have a Convex Pro account, use the Convex Dashboard to file support tickets.

Provide context: What are you trying to achieve, what is the end-user interaction, what are you seeing? (full error message, command output, etc.)
Use search.convex.dev to search Docs, Stack, and Discord all at once.
Additionally, you can post your questions in the Convex Community's #1228095053885476985 channel to receive a response from AI.
Avoid tagging staff unless specifically instructed.

Thank you!

sleek vigil Sep 25, 2024, 4:14 PM

#

currently this isn't possible; npx convex import requires the file to exist on the local filesystem. But @tame nimbus is working on a new feature that should help

lavish comet Sep 25, 2024, 4:14 PM

#

Thanks

#

If I wanted to import manually, can I run npx convex import --replace --preview-name "development" ./data/mock-data/snapshot.zip locally to upload it to a preview deployment whose branch name is "development"? i.e. is the VERCEL_GIT_COMMIT_REF used in the deploy command npx convex deploy --cmd 'npm run build'&&[ "$VERCEL_ENV" = preview ]&&npx convex import --preview-name "$VERCEL_GIT_COMMIT_REF" ./data/mock-data/snapshot.zip;npx convex run --preview-name "$VERCEL_GIT_COMMIT_REF" migrations:runMigrations equal to the branch name? If not, how do I get it?

sleek vigil Sep 25, 2024, 4:15 PM

#

I suppose you could split the file into many files for github and then merge them in the build command.

lavish comet Sep 25, 2024, 4:16 PM

#

Wow yea actually what I did above seemed to work!

sleek vigil Sep 25, 2024, 4:16 PM

#

I believe $VERCEL_GIT_COMMIT_REF is the branch name, yes. Are you trying to import into your dev deployment?

lavish comet Sep 25, 2024, 4:17 PM

#

no, was trying to import into preview environment

#

Just happens to be that my preview environment is off the pushed branch development

sleek vigil Sep 25, 2024, 4:17 PM

#

Nice

lavish comet Sep 25, 2024, 4:23 PM

#

Now that I'm doing it manually, how would I remove it properly from the deploy command but still keep the runmigrations part?
npx convex deploy --cmd 'npm run build'&&[ "$VERCEL_ENV" = preview ]&&npx convex import --preview-name "$VERCEL_GIT_COMMIT_REF" ./data/mock-data/snapshot.zip;npx convex run --preview-name "$VERCEL_GIT_COMMIT_REF" migrations:runMigrations
to
npx convex deploy --cmd 'npm run build'&&[ "$VERCEL_ENV" = preview ]&&npx convex run --preview-name "$VERCEL_GIT_COMMIT_REF" migrations:runMigrations

#

?

sleek vigil Sep 25, 2024, 4:26 PM

#

Do you want to runMigrations after importing the seed data? Then that won't work

lavish comet Sep 25, 2024, 4:26 PM

#

Oh I guess there's no point in running the migration then and there

#

yea

sleek vigil Sep 25, 2024, 4:27 PM

#

As another option, you could store the file in s3 and then curl it to download it before importing

lavish comet Sep 25, 2024, 4:27 PM

#

So just npx convex deploy --cmd 'npm run build' and then locally run npx convex import --replace --preview-name "development" ./data/mock-data/snapshot.zip;npx convex run --preview-name "development" migrations:runMigrations ?

#

As another option, you could store the file in s3 and then curl it to download it before importing
I can do all that in a build command? That'd be cool

sleek vigil Sep 25, 2024, 4:27 PM

#

yep build command should be able to curl

lavish comet Sep 25, 2024, 4:28 PM

#

wow ok will try that

#

Or maybe ill do it in github actions since i can store the snapshot with lfs

sleek vigil Sep 25, 2024, 4:28 PM

#

Ooh good idea

lavish comet Sep 25, 2024, 4:54 PM

#

In the end I was able to put it into an s3 bucket and write a custom build command file that downloads it and then runs and it works!

#

Is there a place where the community can write custom posts to help others? I'd be happy to write this one as an example and share it with the community

sleek vigil Sep 25, 2024, 5:48 PM

#

nice work! you could post in #general or #1141187289024839760 , since that build command sounds useful 🙂

#Importing a snapshot during deploy stored on s3