Llama 3.1 via Ollama

[quaint valve]

You can now use the tutorial on running Ollama on serverless environments (https://docs.runpod.io/tutorials/serverless/cpu/run-ollama-inference) in combination with Llama 3.1.

We have tested this with Llama 3.1 8B, using a network volume and a 24 GB GPU PRO. Please let us know if this setup also works with other weights and GPUs.

[versed wigeon]

Docs on that Docker image are now updated. Thanks for the ping!

[quaint valve]

@versed wigeon thank you very much!

[icy edge]

#1261031850651029524 works with gpu

[runic nexus]

When you say "In the Container Start Command field, specify the Ollama supported model", do you mean literally just pasting the ollama model ID into that field?

[versed wigeon]

Yes. Like orca-mini or llama3.1

[versed wigeon]

Also, the Docker image just updated to version 0.9
pooyaharatian/runpod-ollama:0.0.9

[runic nexus]

I keep getting JSON decoding errors trying to run queries on it...

[versed wigeon]

{
  "input": {
    "method_name": "generate",
    "input": {
      "prompt": "why the sky is blue?"
    }
  }
}

Are you passing this?

[runic nexus]

Yeah:

{
  "delayTime": 117699,
  "error": "{\"error_type\": \"<class 'requests.exceptions.JSONDecodeError'>\", \"error_message\": \"Extra data: line 1 column 5 (char 4)\", \"error_traceback\": \"Traceback (most recent call last):\\n  File \\\"/usr/local/lib/python3.10/dist-packages/requests/models.py\\\", line 974, in json\\n    return complexjson.loads(self.text, **kwargs)\\n  File \\\"/usr/lib/python3.10/json/__init__.py\\\", line 346, in loads\\n    return _default_decoder.decode(s)\\n  File \\\"/usr/lib/python3.10/json/decoder.py\\\", line 340, in decode\\n    raise JSONDecodeError(\\\"Extra data\\\", s, end)\\njson.decoder.JSONDecodeError: Extra data: line 1 column 5 (char 4)\\n\\nDuring handling of the above exception, another exception occurred:\\n\\nTraceback (most recent call last):\\n  File \\\"/usr/local/lib/python3.10/dist-packages/runpod/serverless/modules/rp_job.py\\\", line 134, in run_job\\n    handler_return = handler(job)\\n  File \\\"//runpod_wrapper.py\\\", line 39, in handler\\n    return response.json()\\n  File \\\"/usr/local/lib/python3.10/dist-packages/requests/models.py\\\", line 978, in json\\n    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)\\nrequests.exceptions.JSONDecodeError: Extra data: line 1 column 5 (char 4)\\n\", \"hostname\": \"ogp9bh9fndvgck-64411159\", \"worker_id\": \"ogp9bh9fndvgck\", \"runpod_version\": \"1.6.2\"}",
  "executionTime": 61,
  "id": "c4794910-58f5-4179-98a9-0b0779ba0749-u1",
  "status": "FAILED"
}

request:

{
  "input": {
    "method_name": "generate",
    "input": {
      "prompt": "why the sky is blue?"
    }
  }
}```

[versed wigeon]

downgrade the docker image to 0.0.7

[quaint valve]

I also see this error for 0.0.9, so please use 0.0.8, as that one is working.

I opened https://github.com/pooyahrtn/RunpodOllama/issues/11 to get this fixed.

[quaint valve]

Yes, like this:

[runic nexus]

That works, thanks!

[quaint valve]

Perfect, have fun!

[quaint valve]

Would you mind updating the version in the tutorial as well? And go back to 0.0.8? 🙏

[versed wigeon]

Reverted in the docs

[quaint valve]

Thank you very much!