#production deployment skew

1 messages · Page 1 of 1 (latest)

candid widget
#

can you help me with the following issue:

Staged Production Flow (the problem)

With vercel --prod --skip-domain:

  1. Convex functions deploy to production → immediately live
  2. Next.js build runs
  3. Vercel stages the frontend → NOT live yet
  4. Old frontend is still serving → calling new Convex functions

This creates a mismatch window that lasts until you manually promote. If your new Convex functions have breaking changes (renamed functions, changed arguments, removed queries), the old frontend will error.

lunar ivyBOT
#

Hi @candid widget. I'm kapa.ai, your friendly support bot. I will try to answer your question by searching through the knowledge base. :wave:

#

You’ve run into a real limitation of how Convex + Vercel work when you use vercel --prod --skip-domain: Convex deploys are immediate, while Vercel “stages” the new frontend and only flips traffic when you promote, so there’s a window where:

  • New Convex backend is live
  • Old frontend is still serving → calling functions that may have been removed/changed

The docs and support threads don’t describe a magic flag to make Convex wait for Vercel promotion, and they don’t cover --skip-domain specifically, so there’s no officially documented way to make these two promotions atomic. The recommended mitigation is to structure your deploys and changes to avoid breaking old clients.

Here’s what is documented and what you can do:

1. Use the documented Vercel build pattern

The official pattern is to make Vercel’s Build Command:

npx convex deploy --cmd 'npm run build'

Convex will:

  1. Run your frontend build (npm run build) with CONVEX_URL set to the production deployment.
  2. Only after that, typecheck, bundle, and push functions to production. [Vercel hosting; CLI deploy]

This coordinates the builds so Convex doesn’t deploy if your frontend build fails. It does not change the fact that once npx convex deploy finishes, the new backend is live immediately, while Vercel may still be serving the old frontend until promotion.

So this pattern helps with “build succeeds/fails together”, but not with the “staged then promote” skew you’re seeing.

2. Make backend changes backwards compatible

#

Convex’s production docs explicitly recommend treating backend changes as backwards compatible because users may still be on an old frontend when the backend changes. [Typical workflow]

They call out safe patterns:

  • Add new functions instead of renaming/removing immediately.
  • Add new arguments as optional.
  • Temporarily support both old and new shapes (e.g. unions, optional fields).
  • Only remove old functions/arguments once you’re confident no old clients are calling them. [Typical workflow]

In your staged-prod flow, this is the only fully reliable way to avoid errors during the mismatch window: don’t ship breaking Convex changes in the same deploy that you promote via Vercel. Instead:

  1. Deploy a version where Convex supports both old and new behavior.
  2. Promote the new frontend that uses the new behavior.
  3. In a later deploy, remove the deprecated functions/arguments once you’re sure old clients are gone.

3. Avoid deploying Convex for preview/staged builds (if appropriate)

In a related Vercel preview thread, support suggests that if you don’t want a new Convex deployment for previews, you can simply skip npx convex deploy and point previews at an existing backend (e.g. prod or dev) via a hard-coded URL. [Preview thread]

Applied to your case, one possible pattern (if it fits your workflow) is:

  • For “staged” / --skip-domain builds, don’t run npx convex deploy at all; just build the frontend against the existing production Convex URL.
  • Only run npx convex deploy --cmd 'npm run build' on the final production build that you intend to go live immediately (i.e., no staging/promotion step).
#

This removes the mismatch window because Convex only changes when you’re doing a “real” production deploy, not when you’re staging.

The docs don’t give a turnkey script for this; you’d need to conditionally run npx convex deploy based on environment (e.g. only on Vercel’s “Production” build, not on “Preview”/staged builds). The Discord thread just confirms that “skipping convex deploy altogether” for previews is a valid approach. [Preview thread]

4. Use preview deployments for safer testing (but they don’t solve skew)

Convex preview deployments give you a fresh backend per branch and integrate with Vercel Preview Deployments via:

npx convex deploy --cmd 'npm run build' --preview-run 'functionName'

with a Preview CONVEX_DEPLOY_KEY. [Preview deployments]

This is great for testing, but it doesn’t change the fundamental behavior for production: production Convex deploys are still immediate.

5. Summary of practical options

Given what’s documented:

  • There is no built-in atomic “flip both Convex and Vercel at once” mechanism.
  • To avoid the mismatch window, you can:
    • Design Convex changes to be backwards compatible and remove old behavior only after the new frontend is fully rolled out.
    • Skip npx convex deploy for staged/preview builds and only deploy Convex on the final production build, so staging doesn’t change the backend.
  • Continue using npx convex deploy --cmd 'npm run build' for production builds so Convex and frontend builds are at least coordinated and fail together. [Vercel hosting; CLI deploy]
#

If you describe exactly how you’re invoking Vercel (CI commands, build command, env vars), I can help sketch a concrete command layout that follows these patterns while minimizing skew.

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon:

#

@candid widget Kapa.ai is still learning and improving, please let me know how I did by reacting below.

candid widget
#

@lunar ivy can you see what other resources or information you can share about this. or if anyone else has ran into this problem

lunar ivyBOT
#

Let me check the knowledge base for you again. One moment! :books:

#

There are a few relevant threads and docs, but nothing that fully “solves” the staged-production skew you’re describing. People have run into closely related issues.

1. Very similar report: “Deployment differences between Convex and Vercel”

A user with Next.js + Convex + Vercel reported:

Convex finishes building first — for example, removing a function.
Then, Vercel builds and deploys the React app that still references that function.
During the ~1.5-minute gap … the function no longer exists on Convex, but the frontend still tries to call it. [Deployment differences]

Support’s guidance there was:

  • Make sure you’re using the documented build command:

    npx convex deploy --cmd 'npm run build'

    so Convex only deploys if the frontend build succeeds. [Vercel hosting]

  • But they did not provide a mechanism to avoid the brief window where old clients hit a new backend; instead, they implicitly rely on the “backwards compatible changes” guidance (see below).

The user still reported issues and there wasn’t a follow-up “this fixes everything” answer in that thread. [Deployment differences]

2. Convex’s official stance: backend must tolerate old clients

The production docs explicitly say you should assume old clients will call new functions and therefore:

  • Add new functions instead of renaming/removing immediately.
  • Add new arguments as optional.
  • Only remove old functions/arguments once you’re OK breaking old clients. [Typical workflow]

This is exactly to handle cases like:

#
  • User has an old SPA open.
  • Or, as in your case, the old frontend is still live while the new backend is already deployed.

So the documented answer is: design Convex changes to be backwards compatible across that window.

3. Preview / staged builds: skipping convex deploy

In a preview-deploy thread, support acknowledges that sometimes you don’t want Convex to deploy for non-production builds (e.g. Vercel previews). The suggested workaround:

  • Detect preview builds and skip npx convex deploy entirely, using a hard-coded Convex URL instead.
  • This way, previews don’t change the backend at all. [Vercel preview thread]

That’s not exactly your --prod --skip-domain case, but it’s the same pattern: if you don’t want a backend change until you’re really going live, don’t run npx convex deploy for the “staged” step.

4. How Convex + Vercel are intended to coordinate

The official Vercel integration is:

npx convex deploy --cmd 'npm run build'
  • npx convex deploy:
    • Reads CONVEX_DEPLOY_KEY.
    • Sets CONVEX_URL (or framework-specific env like NEXT_PUBLIC_CONVEX_URL).
    • Runs your build command.
    • Then pushes functions to the target deployment. [Vercel hosting; CLI deploy]

This coordinates build success/failure, but Convex deploys are still immediate once the command finishes. There’s no documented way to “hold” the Convex deploy until Vercel promotion.

5. Other related issues

#
  • A user reported existing clients crashing after deploy because their local frontend was out of sync with the new Convex deployment; support asked for more details but there’s no published solution in that thread. [Existing client crashed]
  • Another user hit errors when using a production deploy key in a non-production build environment; that’s about environment checks, not skew, but shows Convex is opinionated about where deploy keys are used. [Vercel deploy breaks]

6. What’s not in the sources

  • There is no documented feature to:
    • Delay Convex’s production deploy until Vercel promotion, or
    • Atomically flip Convex + Vercel at the same time.
  • There is no example specifically addressing vercel --prod --skip-domain and staged promotions.

So based on the available material, the “official” tools you have are:

  1. Use npx convex deploy --cmd 'npm run build' so builds are coordinated. [Vercel hosting]
  2. Make Convex changes backwards compatible across deploys. [Typical workflow]
  3. For non-final builds (previews / staged), consider not running npx convex deploy and instead point them at an existing Convex deployment. [Vercel preview thread]
#

If you want, you can paste your exact Vercel build command(s) and how you’re invoking vercel --prod --skip-domain, and I can help you sketch a concrete “only deploy Convex on final promotion” setup based on these patterns.

Note: you can ask me a follow up question by @ mentioning me again :speech_balloon: