Inquiry on Utilizing TensorFlow Serving with GPU in Serverless Configuration | Runpod | Page 1

old hazel Aug 1, 2024, 3:33 PM

#

Hello Runpod Community,

I'm exploring options to utilize TensorFlow Serving with GPU support in a serverless configuration on Runpod. Specifically, I'm interested in whether it's feasible to make requests from a Runpod serverless job to a TensorFlow Serving instance running on the same container or environment.

Could anyone clarify if this setup is supported? Additionally, are there alternative recommended approaches for deploying TensorFlow Serving with GPU on Runpod's serverless infrastructure?

Thank you in advance for your assistance!

Best regards,
Sebastian

ebon wharf Aug 1, 2024, 3:41 PM

#

I don't know of anyone doing it today but check out this link, for a starting point: https://www.tensorflow.org/install/docker

TensorFlow

Docker | TensorFlow

open knoll Aug 2, 2024, 5:06 AM

#

old hazel Hello Runpod Community, I'm exploring options to utilize TensorFlow Serving wit...

yes i guess so, if the inference is done via code its easily setup-able in runpod serverless

#

runpod serverless is already inside docker container so you wouldn't want a docker host there

ebon wharf Aug 2, 2024, 9:34 AM

#

There is a template in Explore but it doesn'th have any info: https://www.runpod.io/console/explore/runpod-tensorflow

open knoll Aug 2, 2024, 9:35 AM

#

i guess it comes with tensorflow library preinstalled

#

just like pytorch

old hazel Aug 2, 2024, 8:40 PM

#

Thank you all for the responses and suggestions! I currently have a local setup with a Runpod, TensorFlow and CUDA container that works well but is quite large (~7 GB). I'm also considering using TensorFlow Serving, which could reduce the image size to less than 1 GB. I'll test both approaches and share my findings once I have more details. This might take some time, but I'll keep you posted!
Cheers, Sebastian

#Inquiry on Utilizing TensorFlow Serving with GPU in Serverless Configuration