e2e testing with preview deployments

[unborn zephyr]

I'm trying to get a setup where I can run e2e tests in Github CI against a preview environment. Here's what I have:

      - name: Build with Convex deploy
        run: pnpm convex deploy
          --cmd "pnpm run build"
          --cmd-url-env-var-name VITE_PUBLIC_CONVEX_URL
          --preview-name "e2e-${{ github.ref_name }}"
        env:
          CONVEX_DEPLOY_KEY: ${{ secrets.CONVEX_DEPLOY_KEY }}

      - name: Run Playwright tests
        run: pnpm run e2e
        env:
          CLERK_PUBLISHABLE_KEY: ${{ secrets.CLERK_PUBLISHABLE_KEY }}
          CLERK_SECRET_KEY: ${{ secrets.CLERK_SECRET_KEY }}

I want my tests to run seed functions before they go, e.g.

test.beforeAll(async ({ page }) => {
    await execa("pnpm", ["convex", "run", "seed:characters"], {
        stdio: "inherit",
    })
})

test("navigating characters", async ({ page }) => {
    // etc
})

this doesn't work in CI because the tests are ran outside the convex deploy context and don't have the CONVEX_DEPLOYMENT environment variable

I can --preview-run seed:characters, but I want to be able to run specific functions for specific tests to replicate certain setup scenarios without the test going through the UI to do so

so my question is: how can I pass the info from convex deploy such that I can run functions during the tests?

[unborn zephyr]

although maybe I can approach this differently 🤔 I could hit an API route in my app like /setup/characters, which would then run that function in the backend

and probably checks some conditional PREVIEW=true env to make sure it can't just be run arbitrarily

[unborn zephyr]

here's how I did it for the curious https://github.com/itsMapleLeaf/game-of-arte/pull/34/files

[high lintel]

I was about to suggest something similar to what you have (similar thread -- #1174084317677371604 message)

If we added something like npx convex run "seed:characters" --preview-name "e2e-..." or npx convex run "seed:characters" --url VITE_PUBLIC_CONVEX_URL which required a CONVEX_DEPLOY_KEY, would that work here?

Seems like we need to know which deployment to run the function on (either via the name or via the URL) + some proof that you should be allowed to run (internal) functions on this deployment (CONVEX_DEPLOY_KEY seems easiest since it's static across a project)

[unborn zephyr]

If we added something like npx convex run "seed:characters" --preview-name "e2e-..." or npx convex run "seed:characters" --url VITE_PUBLIC_CONVEX_URL which required a CONVEX_DEPLOY_KEY, would that work here?
yeah that would probably work

although maybe it could be possible to create an authoritative instance of ConvexHttpClient? one that can run internal mutations, given a deploy key

that'd be more straightforward and one less dependency than having to run a child process to run a function

[gaunt flower]

This exists today but is undocumented/unstable, it might be fine to stabilize it. https://github.com/get-convex/convex-js/blob/32ff1dc1190e35223bda44c41f51a237960da3e0/src/browser/http_client.ts#L83

[unborn zephyr]

you have no idea how long I've needed this lmao

[gaunt flower]

It's not likely to go away, a possible behavior change is how it interacts with httpClient.setAuth() since currently you can't use both. The behavior of setting this HTTP header wont' change because we have to support old CLIs.

so you're welcome to use this, worst case is you need to make the HTTP request yourself if we remove it. I don't think we will, we just might clean up some language around "admin keys" vs "deploy keys"

[unborn zephyr]

actually, the issue still remains that I can't get the VITE_PUBLIC_CONVEX_URL variable for the preview after the build step happens, unless I'm missing something? 🤔

[gaunt flower]

Is that sitting in .env.local?

[unborn zephyr]

well, that doesn't exist in CI

what I mean is, in CI, it runs convex deploy ... to run the build and deploy the functions, but I'm not sure how to get the preview URL from that point to run the tests

[gaunt flower]

(adding CLI tools for all these things would be really reasonable, let's say we're just figuring out how to unblock you for now)

after this runs isn't there a .env.local with the new VITE_PUBLIC_CONVEX_URL in it? and this is GitHub CI, so you could grab this for a future step here?

[unborn zephyr]

oh I wasn't aware the deploy command generated an env file

so if it's not being picked up, then that means I'm not loading the env correctly? 🤔

[gaunt flower]

I'm not sure, just wondering how the frontend build step gets this populated normally for a preview deploy

looking

[unborn zephyr]

this is the CI command for build:

        run: pnpm convex deploy
          --cmd "pnpm run build"
          --cmd-url-env-var-name VITE_PUBLIC_CONVEX_URL
          --preview-name "e2e-${{ github.ref_name }}"

[gaunt flower]

ahhh nm nm

ah it just runs the cmd in an environment with that environment variable set so you'd have to build a --cmd that reported the VITE_PUBLIC_CONVEX_URL

like --cmd "env | grep VITE_PUBLIC_CONVEX_URL > saved.txt; pnpm run build"

[unborn zephyr]

ah yeah

alright that seems to work, but there's some other stuff I need to fix on my end that I'll take a stab at later

thanks for the help!