spring pawn
I am trying to run inference on llama3-8b using llama-cpp-python. I want to use the create_chat_completion function which requires me to define a chat_format when setting up the model. For my friend, just setting it to "llama-3" works fine but for me it results in the following error:
Traceback (most recent call last):
File "c:\Users\julia\Desktop\CPPTest.py", line 22, in <module>
response = model.create_chat_completion(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\julia\AppData\Local\Programs\Python\Python311\Lib\site-packages\llama_cpp\llama.py", line 2106, in create_chat_completion
handler = self.chat_handler or llama_chat_format.get_chat_completion_handler(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\julia\AppData\Local\Programs\Python\Python311\Lib\site-packages\llama_cpp\llama_chat_format.py", line 61, in get_chat_completion_handler
return CHAT_HANDLERS[name]
~~~~~~~~~~~~~^^^^^^
KeyError: 'llama-3'
What the hell am I doing wrong?
model = Llama(
model_path=llamaPath,
chat_format="llama-3",
n_gpu_layers=-1,
#n_ctx=2048,
verbose = True
)
print("Model loaded")
response = model.create_chat_completion(
messages = [
{"role": "system", "content": "You are an assistant."},
{
"role": "user",
"content": "How high is the eiffel tower?"
}
]
)
print(response)
zealous zephyr
