frank obsidian
Hi, I tried the example from the official documentation (https://docs.mistral.ai/self-deployment/skypilot), but deploying to Azure instead of AWS. SkyPilot runs without error and gives me an IP address with
IP=$(sky status --ip cluster-name)
Unfortunately, that IP address is not reachable (neither curl nor ping). This seems to be the same issue as in #random message
Any idea on how to proceed?
fallow inlet
Ah, this might be a question for the skypilot folks actually ! I've noticed the same thing and just created an ssh tunnel to the instance
ssh -L local_port:127.0.0.1:remote_port sky_pilot_cluster_name
then you should be able to curl at localhost:local_port
frank obsidian
Could you please elaborate on the ssh tunnel idea?
I used
ssh -L 10000:127.0.0.1:remote_port sky_pilot_cluster_name
to create a tunnel and it seems I'm connected to Azure sky_pilot_cluster_name. But when curling
curl http://127.0.0.1:10000/v1/completions
-H "Content-Type: application/json"
-d '{
"model": "mistralai/Mistral-7B-Instruct-v0.2",
"prompt": "My favourite condiment is",
"max_tokens": 25
}'
I get
curl: (56) Recv failure: Connection reset by peer
PS: I forwarded the initial question to the skypilot folks, see https://github.com/skypilot-org/skypilot/issues/2865
proper pine
if you ssh to the skypilot node, does this curl work? (to 127.0.0.1:8000/v1/completions)
frank obsidian
If I ssh into the skypilot node, then the curl
curl http://127.0.0.1:8000/v1/completions
-H "Content-Type: application/json"
-d '{
"model": "mistralai/Mistral-7B-Instruct-v0.2",
"prompt": "My favourite condiment is",
"max_tokens": 25
}'
does not work, because it results in
curl: (7) Failed to connect to 127.0.0.1 port 8000 after 0 ms: Connection refused
proper pine
looks like vllm is not running properly, do you have logs?
frank obsidian
Unfortunately, I've no idea how to get logs.
But your question gave me an idea: Maybe I can get Mistral running without docker, by setting up vLLM through SkyPilot as in https://docs.vllm.ai/en/latest/serving/run_on_sky.html
autumn tide
If you are still interested @frank obsidian , please feel to check out the mixtral example in SkyPilot repo: https://github.com/skypilot-org/skypilot/tree/master/llm/mixtral
We provide a way to serve the model without using the docker image. : )