#Exposing http ports on serverless

104 messages · Page 1 of 1 (latest)

lofty shore
#

There's no way to expose http ports on serverless is there? When I'm creating a new template and flip the template type from Pod to Serverless that option goes away.

#

I see stuff in the codebase about RUNPOD_REALTIME_PORT but I'm not sure what the use case is for that and I haven't found documentation for it. Also I'd like to expose two ports to my serverless instances

high parcel
#

Why would you want to expose HTTP ports on serverless? Its not designed for that. Use pods instead.

lofty shore
#

I'm creating a version of this https://www.instagram.com/p/C8CzGuOubKp/ that anybody can use without setup. Sending frames over a websocket connection at 24fps.

I have it working with pods, but I'd rather not manage standing up and tearing down the instances for each user

Instagram
sage snow
#

wow that seems cool

lofty shore
sage snow
#

Yeah use pods for now...

#

24fps, isn't that so high, you're gonna use bunch of high end gpus hahah

lofty shore
#

it all runs on one 3090

sage snow
#

Wow 24 image / s?

lofty shore
sage snow
#

Oooo

sage snow
lofty shore
# sage snow Yeah use pods for now...

is there any off the shelf or open source solution for managing a bunch of pods? Ideally I'd like to have some sitting idle for users so they can instantly start using it without waiting a few minutes for the server to stand up and for the start script to run setting up the project etc

sage snow
lofty shore
sage snow
#

kind of insfrastructure as code

lofty shore
#

ah so it's in that land

sage snow
#

use the graphql api if you want to

#

start script i think you can make custom templates

lofty shore
#

the pod version of this is already a custom template. Works amazingly well

sage snow
lofty shore
#

well it's handy for the pod version

#

I think the non-pod version will have to be a pretty different architecture with a whole webapp

sage snow
#

i think for "instant use or faster loading" use some active workers serverless or pods (longer loading if you wanna match the serverless thing )

#

yes unless you got the same architecture in pods and serverless lol

lofty shore
#

yeah that's why I was hoping to have some solution out there for managing the fleet of pods. Obviously for cost reasons I wanna have a few active waiting for people as possible, so it'd have to be dynamic and pretty complex logic of when to stand new ones up/tear old ones down

sage snow
#

make your own hahah, use the graphql api

lofty shore
#

side note currently I'm stuffing the actual models inside of the container which causes my docker images to be like 20 gigs, takes forever to upload and is a terrible developer experience. How are people doing it?

#

why graphql api instead of python sdk?

sage snow
#

well there is an alternative, you can offload the model into network storage

lofty shore
#

can I auto mount a network drive on all new pods I stand up?

sage snow
#

yes, its automatically mounted in /workspace

#

if you choose it in ui or start from graphql, both can be done

#

but you're limited to a region that has the network volume

lofty shore
#

ahh dealbreaker

sage snow
#

dw tho its still a huge pool of gpus, just need to pick the right one that has the gpu model you wanna use

#

but also its slower than container disk

lofty shore
#

well I've noticed some regions aren't good for maintaining the websocket connection. It keeps dropping. and also regions run out of gpus quite frequently in my experience. I can't couple myself to one region, especially not when the current images I have are working fine just huge and annoying

#

tradeoff not worth it

#

wish there were more gpus in usa regions

sage snow
lofty shore
#

most lol

#

RO is working fine

sage snow
#

ooof

#

and us?

lofty shore
#

never had the gpus I want available in US

#

ever lol

sage snow
#

yeah well, gpus are limited from the stock in the DCs

lofty shore
#

DCs?

sage snow
#

datacenters

lofty shore
#

do you work at runpod? you're helpful btw tks for answering my random questions

#

I've felt like I'm wandering through a dark forest lol

sage snow
#

hahaha well not officially working

#

feel free to explore the docs, or find some codes on google, github

lofty shore
#

so I guess my next step is to build the whole dang pod management thing

sage snow
#

also if you're building for serverless you can test out on pods too

#

sure goodluck with that!

lofty shore
#

but geez the serverless stuff all works great I just would love to use that with the ports exposed

#

spin up an job when a user wants to do the live thing, spin it down when theyre done

sage snow
#

why not initiate it from internal to external

#

that seems more fit to pods workload

lofty shore
#

not sure what you mean

sage snow
#

like i said before initiate the ws from the serverless worker, not expose a port then connect from external

lofty shore
#

I think you need the port exposed to make a websocket connection no?

#

hold on spinning up a pod without the ws port open to confirm

sage snow
#

runpod's firewall for outbound is all exposed if im not wrong

sage snow
lofty shore
#

oh snap

#

well nah it needs to go both ways

#

well wait no that might work

#

im gonna quick spin something up locally and ngrok into it and see if that works

sage snow
#

Yeah that might work, try it out on pods if you want

lofty shore
#

ur a genius

#

the only annoying thing is now I need to maintain double the websocket connections

sage snow
#

double?

lofty shore
#

browser -><- my server -><- runpod

#

instead of just sending the frames directly between the runpod instance and the browser

#

in fact the server -><- runpod connection maybe I want to find a better way to do it than websockets. Some messaging queue or something

sage snow
#

sure

#

but that would add some more delay ig

#

if not sockets

sage snow
lofty shore
#

well that's how it works currently

#

wanna try it out lol

sage snow
#

hmm sure

lofty shore
#

dming the link 1 sec

cursive scroll
#

With GenDJ would it be possible to have it process video from a file rather than live? Take in mp4 return mp4?

lofty shore
#

it would totally be possible, just haven't built it yet

#

currently it just turns the frames of your webcam into jpgs and sends them to the server, you'd do the same thing with the video as it plays

cursive scroll
#

I'll take a look at the source, might be able to adapt.

lofty shore
#

PRs welcome lol

cursive scroll
#

Also, for <- my server -><- runpod -> you are not limited to websocket. If it were me I would have servless connect to my_server via a openvpn. Really easy to setup and automate.

lofty shore
#

interesting. I tried cloudflare tunnel but couldnt figure it out. hadn't thought of vpn

#

would that scale? Like many people could be using it at the same time?

#

and how would I send the frames?

cursive scroll
#

Assuming you don't care about hyper focusing on encryption there is very little overhead. You could send frames via SCP or you could mount a disk from the server... If you want to keep it off disk then your could receive it via a webhook.. really anyway you can on network.

#

With that said I am in the process of building out a front end that acts as web socker server and client... sometimes it is the best way to go.

lofty shore
cursive scroll
#

Yes, combine that with openvpn connection and your in business.

#

RunPod has teased about adding backend networks that serverless can connect to and along with your external servers. Until that happens OpenVPN is is pretty close to that.

lofty shore
#

quick question: do we pay for pod startup time?