Google Cloud ML Inference vs. NVIDIA ML Inference
#.
hollow echo
ancient monolith
Do you know exactly what ml inference work you'll do at google? If so, is the work more interesting at google cloud or nvidia?
I'd decide based off that
although if nvidia will alow you to return for ft regardless of whether you do a return internship, taking google is of course the best choice
ancient cloud
It really depends on how close you’d be to the tpu level. I know Google has a tpu inference team where they’re more agnostic to models and just try to expand support to 1p and 3p models. Your day to day is more staring at tensor shapes and optimizing kernels etc
The other end of the spectrum I believe is getting 1p models training and inferencing as quick as possible which I think that skillset is still the same but the day to day I imagine would be different
I can also imagine that being more appealing for frontier labs
Databricks imo isn’t really the best for mlsys unless if you’re in mosaic or their inference teams so you’re right to prioritize google and nvidia
@proud hearth feel free to weigh in with what yk about trt
ancient cloud
I mean it seems you’re knowledgeable enough in this field to have gotten these in the first place. I’m guessing some prior work in ml inference. Specializing in tpu could be a good thing it’s just no one knows yet on whether it’s the right thing yet. Anthro’s philosophy is to just buy all hardware and see which one is best. Their tpu deal was big news but no one mentioned the deal they struck with Annapurna and their chips.
But I think there are many transferable skills from learning xla, hlo, Jax etc
Like if you had to make another runtime like xai did with sglang I think you’d develop the knowledge