#(Beginner Question) Hosting Quantized model

4 messages · Page 1 of 1 (latest)

ancient spade Apr 29, 2025, 6:27 AM

Hi, I'm new to runpod can anyone point me towards how I can host a quantized model like this? I want to try the 2.71bit version first. https://huggingface.co/unsloth/DeepSeek-V3-0324-GGUF-UD

spiral void Apr 29, 2025, 7:02 AM

4xA100
H100

Should work

Try vllm with 4xA100

Its supported with runpod's official template