#Pull Request Environment Variables
merry valeBOT
devout swanBOT
red yarrow
081c7173-adc7-4ae7-a5a3-66a941481611
lone estuary
@past elbow - perhaps you know?
red yarrow
Edit: Third time that my message is being deleted here when editing, not a good use of the bot imo
Tldr for the 3rd time
How do you create an ephemeral PR using github actions where you can dynamically add an environment variable from the github action to the newly created environment? I've been tinkering here https://github.com/Faolain/test-litestar-deployment/pull/1/files but the CLI commands are wrong and I'm not sure quite which ones would do what I need (to create the ephemeral environment and associate the dynamically passed env vars)
lone estuary
you can not edit the links within your message, this is done for safety purposes, please keep this in mind for the future
red yarrow
Fair enough, just didn't want to spam the chat and have it all disjointed, and instead be in one message 😅
lone estuary
what variables would you actually need to have in the pr environment?
theres also no such thing as a railway environments destroy command, so im not sure where you pulled that from lol
red yarrow
I have a dynamic example var as likely some will include 3rd party services that I don't want to spin up in railway (testing serverless solutions), and yeah I'm aware re: nonexistent commands but that's more or less the functionality I'm lookig for (running the cli locally I see there aren't really many commands one can run with environment)
lone estuary
environment Change the active environment
red yarrow
Essentially want a way to ephemerally deploy PRs and pass in env vars from the CI
red yarrow
I want to try NeonDB which can generate and deploy ephemeral DBs which are branched off of production for every PR without affecting the main PR database.
red yarrow
Indeed, in my original post before it was deleted I included this link but it does not show how to do this via github actions
lone estuary
beacuse you dont
red yarrow
There is the ability to turn this on within the Railway Dashboard but nowhere else
red yarrow
Ideally one would be able to create the environment/ephemeral PR directly from the github action in order to be able to dynamically pass vars, in its current instantation one is unable to do so
lone estuary
i still dont know what you need these vars for?
red yarrow
Using outputs that come out of the CI to set things within the project app env vars. As mentioned previously NeonDB offers serverless postgres where you can indefinitely branch a database from another already existing database. In this particular case I want to trial taking that sql endpoint and passing it along to railway.
lone estuary
i see that neon has some workflow scripts that use curl
red yarrow
If a github action was possible to create an environment using the "framework" of this mock github action https://github.com/Faolain/test-litestar-deployment/pull/1/files or if the CLI offered this possibility then it would be relatively trivial. Could create and destroy environments if need be
lone estuary
this is likely what you will also have to do with railway as well
red yarrow
neon has this functionality directly from github actions
something similar would be needed for railway, or if there was an existing CLI command that would offer the ability to create environments. Are you saying for now the environments can be manually created via a CURL?
lone estuary
yes and you would need to use the railway api to make your own action
red yarrow
gotcha makes sense , so assuming the question moreso here is that is this offered via the railway api natively?/ any documentation around this?
lone estuary
the neat thing about railway's api is that if you can do it in the dashboard, you can do it with the api
red yarrow
appreciate it, a little more involved than I would have liked but I guess it's the only solution. Any idea if there's a roadmap, issue board, or a canny where features can be requested/upvoted? Adding this functionality to the CLI for environments create/destroy seems like a good candidate 😅
lone estuary
environmentCreate
environmentDelete
variableUpsert
may be some of the mutations you will need
to be frank, this is a relatively niche feature request so keep in mind this may not ever make it into the cli unless you wanna do a pr for it! dont mean to be a downer just being honest
https://feedback.railway.app/
red yarrow
yeah no worries at all, really appreciate the insight/assistance as always
lone estuary
no problem!
red yarrow
Question, I'm looking at building the github action from scratch to deploy PRs using the graphql api and I was able to create an environment and this shows up which is so far fine. However the services do not include any environment variables, despite providing a sourceEnvironmentId (which it did copy all the same litestar services redis/postgres/and the litestar app) but not the env vars.
I see if one does the "enable PR environments" (for auto pr deployments) you can set the env vars to use as the base environment. Is there something similar that can be done on environment creation/update from the graphql api?
this is the graphql query Im using (where the idea is I will ultimately take that domain and pass it over to vercel)
mutation {
environmentCreate(input: {ephemeral: true, name: "Insomnia Test", projectId: "081c7173-adc7-4ae7-a5a3-66a941481611", sourceEnvironmentId: "7fc1ebb6-be04-493a-becf-572178bb297b"}){
id
createdAt
serviceInstances {
edges {
node {
id
domains {
serviceDomains
{
domain
id
}
}
}
}
}
}
}
Is the only option to chain a query variables(environmentId: String! pluginId: String projectId: String! serviceId: String unrendered: Boolean): ServiceVariables! and pass the variables to a mutation of a variableCollectionUpsert?
lone estuary
environmentCreate should copy the variables too, because thats what the web ui calls
only thing is that it doesnt set ephemeral to true
red yarrow
it doesn't look like it does, I created that environment using sourceEnvironmentId and it didn't copy
Empty Variable Set
lone estuary
you should really be using the query variables area
red yarrow
Production (where the test PR was set from) Variable Set
red yarrow
you should really be using the query variables area
Not sure what you mean?
lone estuary
you are hardcoding the variables into the query, use this section
but either way, i will do some tests with your query to see if i can replicate it not copying variables
red yarrow
Those aren't the env variables for the query
That's from insomnia https://docs.insomnia.rest/insomnia/graphql-queries
red yarrow
I guess I don't understand then why the env variables aren't being copied but everything else is then during the environment creation?
lone estuary
this is postman, same thing though
this is how you should be doing things in insomnia too
red yarrow
this just allows you to dynamically pass the vars but that's not the issue here right? I'm hardcoding the values at the moment for testing
lone estuary
no its not, im just saying
red yarrow
gotcha
yeah in the future I'll be passing them dynamically but since I'm just testing with one project I don't mind it being static atm
lone estuary
bad habit is all
red yarrow
fair enough
lone estuary
im working on the query though dw
red yarrow
appreciate it thanks Brody! Also I wanted to mention you said you're not on the railway team but you seem to work for them? unless this is volunteer?
lone estuary
just volunteer
red yarrow
Well you've done a tremendous amount of assistance for onboarding people into railway/triaging their problems, who does one dm/tag from railway to get you hired 😅
lone estuary
haha im not qualified
red yarrow
that's not true
lone estuary
it is, trust me
red yarrow
Well I still do extremely appreciate all of your assistance and do think you'd be a valuable member/addition to their team.
lone estuary
thank you 🙂
mutation environmentCreate($input: EnvironmentCreateInput!) {
environmentCreate(input: $input) {
id
createdAt
serviceInstances {
edges {
node {
id
domains {
serviceDomains {
domain
id
}
}
}
}
}
}
}
{
"input": {
"name": "",
"projectId": "",
"sourceEnvironmentId": ""
}
}
passing ephemeral true will not copy service variables, omitting that will copy the variables
red yarrow
interesting thanks Brody, hmm it won't theoretically pop up then as PR on the dashboard right?
lone estuary
i dont see any visual difference on the dashboard either way, only difference i can see is environments created with ephemeral: true don't have service variables
red yarrow
can you send me a friend req? I'm unable to dm/send one myself
lone estuary
sure!
red yarrow
I'm almost there (so sorry again) but making progress on this action. I'm attempting to update the environment variables for the created service and when inputting the variables in the seemingly provided exampleExample: "{ foo: 'bar', baz: 'qux' }" manner I get some strange behavior where it seems the the graphql server is splitting up the variables by string and inserting each as an individual row? Probably something silly I'm missing here.
lone estuary
oh ive seen this before
unquote the variables object
then youd probably need to quote the variable key too
red yarrow
lone estuary
key and value needs to be quoted with the double quote
red yarrow
Had tried that too previously
{
"input": {
"environmentId": "b2cc9816-5fe3-4f36-b8b4-e26e04efb0bc",
"projectId": "081c7173-adc7-4ae7-a5a3-66a941481611",
"serviceId": "43327ddc-56e8-4fbc-ac45-421af23582cc",
"variables": {"DATABASE_URL": "test_url"}
}
}
Felt like I tried every possible combination from the provided example, everything wrapped in quotes/no quotes/single quotes/double quotes for keys/values etc
lone estuary
one sec, ill figure it out
mutation variableCollectionUpsert($input: VariableCollectionUpsertInput!) {
variableCollectionUpsert(input: $input)
}
{
"input": {
"environmentId": "",
"projectId": "",
"serviceId": "",
"variables": {"DATABASE_URL":"test_url"}
}
}
insomnia will complain about a type, but it still works
red yarrow
ah okay appreciate it Brody, I previously had it set but didn't even bother sending bc of the type error thanks again!
lone estuary
test it before thanking 🤣
red yarrow
it worked haha
lone estuary
woohoo
timid ether
Hey @red yarrow - sorry about that, Discord's been going through an abuse wave so we had to block link editing.
red yarrow
Went on a bit of a hiatus from this but back, I was wondering @lone estuary or anyone else for that matter if workers should be spawned as a service? Right now on the railway setup you did brody you just run the worker jointly via run-all but could one run the worker separately as a different service (much like the docker-compose file indicates) https://github.com/litestar-org/litestar-fullstack/blob/main/docker-compose.yml and if so, is there a way within railway to indicate a low bound (including 0) and an upper bound by which this service process can scale to?
lone estuary
you could totally run the worker and app in separate services, as long as they don't need access to the same filesystem
and the service is free to use as little or as much of the resources as possible, no configurations needed or are even possible
red yarrow
yeah for the purposes of this template don't think it would need access to same filesystem (since I don't believe it does anything with the fs, nor do I think it should be encouraged). And interesting re: services using as little or as much of resources possible, my questions is more around how many workers to spawn within a service dynamically. Is there any way to use the API to set that? or dynamically scale based on some metric? The reason I ask is because I believe fly.io provides users the ability to set a "lower bound" and optional upper bound for "machine" scaling, which isn't quit the same but acts as a sort of proxy
Basically is there a way with railway to say hey after a queue is x amount, spawn another worker within the worker service? and once the queue is < a certain amount spin it down.
lone estuary
there's not, this is something your code or litestar needs to manage
you do get 8 vcpu so I don't even know if litestar can make use of multiple cpus
this is multiple cpus, not multiple cores btw
red yarrow
Thanks Brody, and hmm so it would have to be done on the app layer(?), as for litestar this is moreso a worker thing than litestar itself and particularly was thinking about this #1144691393357434980 message when it came up solving the initial setup issue wherein run-all was spawning the max number of http workers for available vCPUs (32+1) wherein you swapped it to be a fixed number. in order to mitigate the Memory usage. But I was thinking if workers were to be on a separate service then how could this be dynamic based on some external metric (queue size for example).
Tl;dr dynamically scale workers independently from the app on a separate service, seems like the suggestion is to do the scaling on the app layer vs the infra (Railway) layer. Seems like the solution to keep things simple and effective on this setup (was pondering if the github action initially should do anything here for the service side) is just to scale worker processes on the app layer if need be and don't separate into a different railway service for now.
lone estuary
I don't really see the point in auto scaling, you have fixed max resources on the hobby or pro plan
red yarrow
Autoscaling for more intelligent/efficient resource consumption really
lone estuary
just seems like something at framework level, nothing to do with railway
red yarrow
Fair enough, sometimes one can do these things from an infra/platform level only reason I asked
lone estuary
the only thing railway would be able to do would be to add more replicas, but unfortunately they don't do that automatically yet
red yarrow
makes sense thanks brody, should have this github action ready for review at some point soon, will have a create (which entails update as well within it) and a destroy action for preview environments
lone estuary
awsome!
red yarrow
Slight hiatus but back, I noticed that the github action is failing which I've been able to replicate with insomnia.
Turns out there appears to be some limit on names when creating environments?
lone estuary
yes 32 characters long
red yarrow
ah gotcha, I meant moreso on the api docs, maybe a length param can be set on the graphql endpoint?
lone estuary
or just better error responses from the api lol
red yarrow
that too would work haha (re: better error responses)
any suggestions on this for PR names? since a PR could in theory have a long name + number + sha
is there any possibility that names can be extended in length?
lone estuary
oh yeah there could be comment text added to inputs
I still don't work for railway you know
red yarrow
ah :/ fair, anyone who does who could see this?
lone estuary
it's the weekend 🙂
red yarrow
since even if I were to use the shortened sha of 7 pr_name +pr number + sha7 could easily/eventually break
lone estuary
honestly i think the 32 char limit is very reasonable
there is definitely ways you can keep the name under that
red yarrow
if only using pr-{number}-{sha7} I guess, will just do that in the meantime then
would've liked to have the name in it, but so it goes
lone estuary
why do you need the sha?
do the environments railway creates from pr's have the sha?
red yarrow
more of a standard within CI systems to determine unique branches and such
red yarrow
do the environments railway creates from pr's have the sha?
not that I'm aware of , but the manual ones I am creating for PR previews with the github action need something to identify them
and to me a name isn't sufficiently unique
but that's okay
lone estuary
wouldn't having the pr name in the environment name mean you are accepting user input, even if the team did bump it to 64 chars you could still theoretically exceed it
red yarrow
this is true but 64 is more reasonable for something like that, I think pr-number-sha(7) should be a good balance though without needing the name, slightly use some descriptive ability within the dashboard but I think it's ok
lone estuary
sounds like that is the right away to go about the name regardless of char limits
red yarrow
This doesn't seem quite right, just not sure of the order/what needs to be called
It correctly creates a new env pr-4-525bdc9 on the Railway Dashboard with the correctly submitted environment variables
Action: https://github.com/Faolain/railway-pr-deploy/blob/main/index.js
Action in Use: https://github.com/Faolain/test-litestar-deployment/pull/4/files#diff-944291df2c9c06359d37cc8833d182d705c9e8c3108e7cfe132d61a06e9133dd
Errors Out in GH Actions with: https://github.com/Faolain/test-litestar-deployment/actions/runs/6142684947/job/16664821057?pr=4
Error: Action failed with error: Error: Only a single deployment trigger is allowed per (environment, service) pair: {"response":{"errors":[{"message":"Only a single deployment trigger is allowed per (environment, service) pair","locations":[{"line":3,"column":13}],"path":["deploymentTriggerCreate"],"extensions":{"code":"INTERNAL_SERVER_ERROR","exception":{"name":"UserError","status":400}}}]
Which I assume has to do with me creating an envrionment but then also creating a deployment trigger. I simply want to deploy a new "fork" of the existing service (preserving all the same configuration/environment variables etc) and instead just want to deploy the new branch with new variables and the code from the new branch I'm deploying from. Not sure what I should be removing in order to achieve this.
A deploymentTriggerCreate is maybe the wrong thing to do, I don't believe I want to create a trigger but I don't see any other option that includes the action to "make a deployment with this branch" as an input.
past elbow
Hey, could I get a tl;dr on what you're trying to accomplish with this setup?
Seems this thread has various different discussions in it & I'm not sure what's what 😅
red yarrow
Hey, could I get a tl;dr on what you're trying to accomplish with this setup?
S...
Hey @past elbow just creating a GitHub action which deploys an ephemeral PR environment instead of relying on the native PR deployments railway has. The reason for this is that I want the ability to switch out environment variables for preview environments based on other GitHub actions.
red yarrow
youd want to do what railway up does
hmm I'm not quite sure I follow here (apologies for this) , so should I replicate this instead or is there an equivalent graphql mutation within the https://railway.app/graphiql# endpoint docs that does this same thing?
And should I replace the line where I run deployService https://github.com/Faolain/railway-pr-deploy/blob/main/index.js#L210 with the call used in https://github.com/railwayapp/cli/blob/master/src/commands/up.rs#L226 ?
Keeping everything else the same, i.e. the environment creation and such?
lone estuary
upload the code to the service with the endpoint used in Railway's cli, this way you won't get any errors about sources being the same
red yarrow
Understood my question is moreso, is the rest of the action correct then? i.e. wherein I create an environment for the pr if it does not already exist and update the environment variables?
So the next step would be use the same endpoint as railway up? If so, where do I specify what branch (the pr) of the code to use? (the graphql mutation does indicate this for deployTriggerCreate but I don't see this being passed in the payload in this railway up endpoint) Should the service created within the environment be modified in some way to include this branch name?
lone estuary
honestly I think I'm a little out of my depths here as I've never used their api to do something like this
red yarrow
fair enough, really appreciate your help here Brody. Maybe @past elbow or someone else might know better?
red yarrow
Tldr for anyone from railway reading this
Trying to create preview environments from a GitHub action based on an existing environment wherein I can set particular environment variables for that deployment. I went ahead making a GitHub action since it seemed like it was the only way to accomplish this but I'm having trouble with the final step to deploy newly created environment based on a new branch as I'm not exactly sure what endpoint to use (the cli appears to differ from the graphql endpoints provided)
past elbow
Here's how I'd do it using GHA:
Setup:
environmentCreate, providing it asourceEnvironmentId(the base env you want the new env to fork from)deploymentTriggerCreate, providing it the envId (and other required info) of what you received from the prev mutationvariableCollectionUpsert, providing it the environmentId and variables
Teardown:
environmentDelete, giving it the ID of what you received- Might be tricky but GHA has events you can filter on, so I'd add this as part of your main workflow listening for "pr merged" events and tearing down the environment where appropriate
Does that make sense? If your implementation differs from the layout above, could you share specific details on how/why?
oh I think I see where your error's coming from
red yarrow
If it's helpful
It correctly creates a new env pr-4-525bdc9 on the Railway Dashboard with the correctly submitted environment variables
Action: https://github.com/Faolain/railway-pr-deploy/blob/main/index.js
Action in Use: https://github.com/Faolain/test-litestar-deployment/pull/4/files#diff-944291df2c9c06359d37cc8833d182d705c9e8c3108e7cfe132d61a06e9133dd
Errors Out in GH Actions with: https://github.com/Faolain/test-litestar-deployment/actions/runs/6142684947/job/16664821057?pr=4
Error: Action failed with error: Error: Only a single deployment trigger is allowed per (environment, service) pair: {"response":{"errors":[{"message":"Only a single deployment trigger is allowed per (environment, service) pair","locations":[{"line":3,"column":13}],"path":["deploymentTriggerCreate"],"extensions":{"code":"INTERNAL_SERVER_ERROR","exception":{"name":"UserError","status":400}}}]
Which I assume has to do with me creating an envrionment but then also creating a deployment trigger. I simply want to deploy a new "fork" of the existing service (preserving all the same configuration/environment variables etc) and instead just want to deploy the new branch with new variables and the code from the new branch I'm deploying from. Not sure what I should be removing in order to achieve this.
From a prior message *
past elbow
Can you try removing the deploymentTrigger logic? We create a trigger when you call environmentCreate, so it'd error out since the trigger already exists
Or even better, I think if you pass in the ephemeral flag to environmentCreate, it should function the same as regular ephemeral envs on Railway - meaning we handle the teardown etc. for you
Effectively that's:
const env = await environmentCreate(..., ephemeral=true)
await variableCollectionUpsert(environmentId=env.id, ...)
and nothing else (no deployService etc.)
It should work if your fields are correct + has the right repo permissions
red yarrow
Or even better, I think if you pass in the `ephemeral` flag to `environmentCreat...
So the issue as to why I removed ephemeral=true (whcih I initially had) was that if ephemeral=true it clears out all the environment variables
Link to Brody's message above where noted that
As for the teardown since I'm manually creating this, I'm not sure how that automatic teardown could even work (aka having railway handle it for me) For example how would it be triggered? For what it's worth I don't mind having the github delete action (which I've also already created) the issue is moreso getting an preview environment deployed which reads from the correct branch.
The reason I added deployService was because using environmentCreate I wasn't sure how to ensure that the environment deployed was the correct branch instead of just main
past elbow
“clear out” meaning the new env’s service starts from blank slate?
red yarrow
yep correct there are 0 environment variables if you set ephemeral: true
not sure if that's intended but that's what ends up happening
past elbow
Did you specify a source env?
red yarrow
yep
if I remove ephemeral: true they all autopopulate normally based on the srcEnvironment
that said I got around that issue without setting ephemeral: true so my only remaining issue is how to set the branch if I don't use deployService. What's the recomended way to set the deploy branch using environment create? Because based on this there is no branch that can be passed in
past elbow
brody knows more than I do
lone estuary
that cap
past elbow
That behaviour we want here is apparently locked behind this setting, which is locked behind PR environments, which you don't want to enable here since it'd just create dupes
.....eek
red yarrow
Yep exactly
Which again it's fine and tbh even preferable if they were separated as the current setup "almost" works
past elbow
What's the recomended way to set the deploy branch using environment create?
Try usingmutation.deploymentTriggersonenvironment.DeploymentTriggerto update the branch/repo
past elbow
yep
red yarrow
Will try first using graphql directly and report back, then if if it works will modify the action (as that's a bit more of a pain)
Won't be home for another hour and change so won't know until then
past elbow
lmk if that works! Meanwhile I'll see what I can do to make your life easier. I think the hoops you had to jump through here is less than ideal for something so simple 😕
lone estuary
simple, I'm still confused
past elbow
environmentCreate creates a deploymentTrigger so you can't call it again (1:1) -> there's no way to specify the settings for deploymentTrigger when that happens -> can't deploy the branch
more broadly for context - sadly the API/docs is very bare when it comes to implementation details like this (a large part of why it's still priority boarding-only), there's a bunch of business logic behind the primitives that we don't expose and when you try to "imitate" what we do under the hood to make all that tick, you'll notice some pieces just don't make sense right now
past elbow
That behaviour we want here is apparently locked behind this setting, which is l...
though I wonder if you can use the API to just enable this specific setting w/o even having "native PR environments" enabled. That's the prForks field on mutation.projectUpdate if anyone wants to try it 😛
red yarrow
Tested on insomnia deploymentTriggerUpdate after environmentCreate worked to automatically update the PR within railway :), that makes life slightly easier not needing to manage the "synchornization" aka when a commit is pushed to the PR as I had assumed I would have to take care of that too
does anyone know if I close this PR if it will automatically tear down the environment or will I have to do this automatically? Gonna try now and see what happens. Edit: Closed PR nothing happened to the environment as expected with github actions turned off (was hoping for a pleasant surprise but this is still ok since I already have the action written). Also probably expected behavior, when you close a PR and reopen it, and push to it, it will still update the environment.
red yarrow
Updated the Github Action looks like it works 🙂(err I think)
The deployment on the dashboard still points to main
but settings points to the right one
For reference this is the
Github action: https://github.com/Faolain/railway-pr-deploy/blob/main/index.js
Github Workflow: https://github.com/Faolain/test-litestar-deployment/actions/runs/6154321506/job/16699515312?pr=6
I assume what's happening is that environmentCreate, is creating the environment based on main (the srcEnvironmentId) and then the update happens after, so any subsequent updates to the PR are reflected in the branch but the idea is that the initial PR should be deployed without needing a second trigger commit
lone estuary
all going over my head
red yarrow
idts, you've helped me with far more complicated things haha. I think in this particular case it's just specific to how railway works in the background so it's a bit unknown but tl;dr
- const createdEnvironment = await createEnvironment(srcEnvironmentId);
-
const { id: deploymentTriggerId } = createdEnvironment.environmentCreate.deploymentTriggers.edges[0].node; // Set the Deployment Trigger Branch await deploymentTriggerUpdate(deploymentTriggerId); -
// Get the Service ID const { serviceId } = createdEnvironment.environmentCreate.serviceInstances.edges[0].node; // Update the Environment Variables const updatedEnvironmentVariables = await updateEnvironment(environmentId, serviceId, ENV_VARS);
is seemingly not in the right order or something needs to happen in order to prevent createdEnvironment from immediately deploying upon being called or to somehow set the branch via deploymentTriggerUpdate before the environment is created?
I guess we can just wait for @past elbow to review and see what the option can be
a hacky way I would imagine would entail somehow getting the current deployment cancelling or removing it and...then calling this?
environmentTriggersDeploy(input: EnvironmentTriggersDeployInput!): Boolean!
Deploys all connected triggers for an environment.
but this seems both brittle and convoluted/unsure if it would work
red yarrow
if I had known it would be this much I definitely wouldn't 😅 but sunk cost fallacy and it's basically almost done so at this point just trying to push it across the finish line. It deploys, points to the right branch, and on subsequent PR commit pushes it works, just need it to deploy initially from the right branch and it's done
lone estuary
if I had known it would be this
well i could have told you it wasnt worth it lol
red yarrow
I've been looking at PaaS's and since railway was already setup with litestar I figured it couldn't be that hard right? After all ephemeral/preview PRs exist across all such as Render Fly.io and others but I didn't want to bother getting the litestar framework setup with them especially since we already have a template to deploy them. Now we're almost there just need to wait on @past elbow I guess for the final cherry on top
lone estuary
ayo please chill with the pings
past elbow
though I wonder if you can use the API to just enable this specific setting w/o ...
Hey! Sorry, super bz day.
^ is last thing I can think of to make this work for you right now. If you call it once it should enable it for the project, but I’d test it out on a fresh proj to make sure nothing goes wonky. It’s an unsupported path to be clear, but it’ll unblock you from using the ephemeral=true approach if it works
Short of that… There’s no way for you to do this right now. Sorry for the wild goose chase here, I’ll try to slot in a patch to make it work with ephemeral=true without the workaround above. No promises on timeline sadly
red yarrow
Hey! Sorry, super bz day.
^ is last thing I can think of to make this work for ...
thanks Ray so to be clear here are these the steps that should be tried?
// Updates project to allow for {ephemeral: true } aka PR support
- projectUpdate
// Create Environment with ephemeral: true
- const createdEnvironment = await createEnvironment(srcEnvironmentId);
how would I set the branch for that environment? In other words here https://github.com/Faolain/railway-pr-deploy/blob/main/index.js#L183, how will it know which PR to deploy
ideally if there was an endpoint or a parameter that would allow to set the branch upon environment creation then it would be solved no? As right now almost everything works except for the initial commit deploying the original main, all subsequent PRs work (since the deploymentTriggerUpdate ensures that the new branch is the source)
past elbow
parameter that would allow to set the branch upon environment creation
yep exactly!
// Updates project to allow for {ephemeral: true } aka PR support
this is a one-off, you don't need it in CI, and just for testing out because I have no idea if that's going to work 😛
red yarrow
I guess my question is more along the lines of how does this help/what should I be testing 😅
/how to confirm if it works (even without the CI)
so if I change this manually via insomnia for my project I can set ephemeral: true for the subsequent CI runs but I'm not sure how that helps (since there is no way to pass a branch to the environmentCreate right?)
past elbow
So if you try that prForks=true tweak on your project, it'd solve the "ephemeral=true creates an environment without any variables" issue, because it'd fork the variables from your base branch, if it works
However, I have no way of telling if that'd work for you, but if it does you can just use ephemeral=true to do all of this without having to care about deploy/triggers etc.
red yarrow
the "without any variables" issue would indeed be solved (which fwiw is solved atm just by doing the environmentUpdate) but then how would I set the branch it should build from? Since I still need to point it to build from the new PR branch no? (Hence the deploymentTriggerUpdate)
the issue atm is that there is seemingly no way to set the branch to build from (aka the PR which triggered the environmentCreation) without an update after the fact, let me know if I'm not being clear
past elbow
ya, so that's impossible right now. I re-checked and the hack I suggested wouldn't work as well because you'd run into the same issue of having it deploy from the initial base branch and needing to update the trigger, so your initial deploy would still be based off the base branch/main
red yarrow
hmm :/
past elbow
out of curiousity, what envs are you smuggling into the environment from GHA?
red yarrow
external services such as databases like upstash or neon
lone estuary
watch there be a native way to do that
red yarrow
which allow for conditional branching off of dbs too (branching mechanisms for the db are very useful/modular)
past elbow
would those envs be branch-specific?
red yarrow
correct
a neon postgres branch per PR
(wherein it branches off of prod, so you can use prod data within a branch without affecting the prod data itself)
lone estuary
I feel like there's something you could do in code for that
red yarrow
the code would be the github action (which both neon and vercel have) the issue here is railway doesn't
past elbow
is that just a connection string per-branch?
red yarrow
is that just a connection string per-branch?
indeed it would be a connection string/user details passed in from the github action from neon completing its creation as the GHA output
and then ideally passed in as the input as an env var for railway wherein the output (api domain) is passed in as an input to Vercel (or any other frontend host that matter)
past elbow
can you read that neon-provided string as part of booting up your application in a GHA step?
red yarrow
the booting up happens on railway though no?
via the building/running of the app?
aka reads the postgres env var (among other vars) and builds/deploys the app using those env vars
past elbow
wait, so your CI here really is just for passing through variables?
red yarrow
indeed that's all, it's just to chain along some actions and orchestrate the creation of multiple services
Idea being something like this
name: Deploy Pull Request to Ephemeral Railway Environment
on:
pull_request:
types:
- opened
- closed
jobs:
deploy:
runs-on: ubuntu-latest
if: github.event_name == 'pull_request' && github.event.action == 'open'
steps:
- name: Checkout code
uses: actions/checkout@v2
...
- name: Create Neon Branch with GitHub Actions Demo
run-name: Create a Neon Branch 🚀
jobs:
Create-Neon-Branch:
uses: neondatabase/create-branch-action@v4
with:
project_id: rapid-haze-373089
parent: dev
branch_name: from_action_reusable
api_key: {{ secrets.NEON_API_KEY }}
id: create-branch
- run: echo project_id ${{ steps.create-branch.outputs.project_id}}
- run: echo branch_id ${{ steps.create-branch.outputs.branch_id}}
- name: Create PR environment on Railway
if: github.event.action == 'opened'
uses: Faolain/[email protected]
with:
RAILWAY_API_TOKEN: ${{ secrets.RAILWAY_TOKEN }}
PROJECT_ID: ${{ secrets.RAILWAY_PROJECT_ID }}
SRC_ENVIRONMENT_NAME: production
DEST_ENV_NAME: pr-${{ github.event.pull_request.number }}-${{ env.INITIAL_SHA }}
PROVIDER: python
ENV_VARS: '{"database_url": "${{ steps.output-branch-id.outputs.branch_id }}"}'
branch_name: ${{ github.head_ref }}
repository: ${{ github.repository }}
I think I figured out a way to do this
Done programatically via Insomnia, requires 2 ish more steps to do in the github action
it's not ideal (having to wait aka poll for initial deployment to finish in order to update the trigger and redeploy the service) but it works
lone estuary
no good
is there some type of environment clone thingy that clones the environment minus deploying the services?
red yarrow
I could be wrong but after poring over this for a bit https://railway.app/graphiql# I didn't see anything like it, would be useful though. A few ways to skin a cat so to speak but yeah ideally I think environmentCreate(...) should just have an optional branch param to pass in which would solve all this roundabout jankiness. *Seems simple enough to add especially as an optional parameter it wouldn't break anything (famous last words I know) but in the grand scheme seems like a useful thing to have (if branch param present set that as the branch instead of the original srcEnvironment branch)
lone estuary
there's really no way to duplicate an environment without the new services having deployments?
red yarrow
I could be wrong but it didn't seem like it 🤷♂️ so there will have to be a superfluous initial deploy to get this setup in the meantime
the first deploy to create the environment and a subsequent deploy to redeploy to build from the correct branch after correcting the deployTrigger
lone estuary
I like my idea better
red yarrow
if that exists I'm all ears haha
lone estuary
have your code create a neon branch and use the resulting database url
let railway do its own pr environments
red yarrow
the code shouldn't be aware of any infrastructure decisions
as that's tying the code to infrastructure
especially in the case of this template framework the code should be PaaS agnostic
lone estuary
faolain can have a little vender lock in as a treat
red yarrow
Prob wouldn't get merged upstream if I did that 😛
lone estuary
okay fine, have a pre-run script
yes that's still coding for infrastructure, but it would not touch any app code
red yarrow
in any case I think my solution works for now aside from the superfluous initial deploys, maybe Ray can chime in tomorrow to see if this seems like a viable approach or if there are ay footguns I should worry about. Don't need to run any code and just touch the github workflow file which is clean and separates the concerns nicely, at the end of the day the app just cares about getting a database_url doesn't care where it comes from or how it gets it.
lone estuary
what happens when you set empherial to false and don't give it a source environment id?
red yarrow
ephemeral has been false (I never set it to true so I assume default is false), as for not giving it a sourceEnvId not sure but if I had to guess it wouldn't copy any of the env vars but I can try now and see what happens
lone estuary
copy them yourself and replace with neons branch environment variables as needed
even if it makes an empty environment it wouldn't be hard to copy all services over manually and set the variables you need
red yarrow
just tried, not sure why I would do this?
it does the same thing as my existing one except it doesn't include any of the env vars
lone estuary
but does it have deployments?
red yarrow
yep
lone estuary
that's no fun
okay hear me out
empherial projects
you definitely can create blank projects
red yarrow
but why would I do that? The solution already exists it just entails an extra deploy which I think is a reasonable stopgap
creating a blank project separates the environments that should be grouped and makes management even harder so that's not preferable
lone estuary
fair
but how do I say this nicely
I wouldn't personally ever accept the solution you have come up with, I'd watch for something official to be implemented in the api
red yarrow
that's understandable but there appears to be no other working solution in the meantime right?
lone estuary
and I'm not saying that because you could have done better, I'm saying it because that type of thing would bug me so much
I'm the type of person to make sure my service doesn't redeploy until I absolutely need to it, I press the cancel button so much
red yarrow
haha dw I know what you mean and fwiw it bothers me too haha but one also has to be pragmatic, modular ephemeral environments are definitely asked for a bunch (I did quite a deep search for render.com/fly.io and a few other PaaS) So many forum posts of people asking for similar functionality funnily enough with railway exposing the graphql in the way that it has, it has given enough tools to at least get a working solution which other PaaS can't say.
lone estuary
who did you say does support this exact same thing? vercel?
red yarrow
I think this is testament to giving the users the tooling via an API like railway has to allow them to build what they need
red yarrow
who did you say does support this exact same thing? vercel?
yeah vercel does but I understand they have been around for a while and are well capitalized/have a mature ecosystem
but they also don't support API service deployments and only support serverless stuff/static frontends, so that's their tradeoff, everyone is missing one or more things 😅 which is okay not everyone can do everything perfectly (and at a reasonable price!! for example Vercel has neon and upstash integrated but adds significant surcharges vs using the tooling natively)
lone estuary
is vercel kv just upstash redis or something?
lone estuary
and what's the benefits from using vercel kv over upstash redis
red yarrow
but the pricing is significantly higher than if you were to just integrate on your own...I understand they want to make some money but they reduced offering substantially (90% reduction in free tier for example among other things)
price
lone estuary
wdym I thought vercel kv is more expensive than upstash redis
red yarrow
yeah it is sorry I misread, the benefits for using it via vercel is less setup
no need to pass urls through like this discord thread is about
"native integration" aka less hassle it just works
lone estuary
interesting
red yarrow
which doesn't make much sense considering it's the same exact service in the background on Vercel's end
lone estuary
so how does that even work? does vercel own neon and upstash and are running those services themselves?
red yarrow
own them how? They are separate companies, but as to how it works in the background no idea ultimately but Vercel Postgres is Neon, and the KV is Upstash. If Vercel is hosting it themselves or just piping it in the background to the 3rd party I'm not sure. If I had to guess maybe the latter? Nevertheless for example I would love if Railway could integrate these services then it would be one less thing users would have to worry about 😅
lone estuary
if vercel isn't running those services themselves wouldn't there be added latency from forwarding the requests from vercel apps to upstash or neon?
red yarrow
not sure maybe they have private networks/same regions deployed, actually now that I write this I think that's exactly it since I remember reading somewhere you needed to deploy to a certain region at the time to be able to use one of them
but yeah informal integration request if railway can integrate neon as a managed postgres service 🙏 lol
red yarrow
in any case in the meantime I'll get this github action working with the superfluous janky deploy and wait on Ray for the blessing if this is the recommend right approach, also in the meantime appreciate the assistance and moral support as always Brody.
lone estuary
moral support 🤣