#Using the example chat template for OLlama raises RuntimeError

13 messages · Page 1 of 1 (latest)

eternal quest
#

I'm trying to fine-tune "unsloth/Meta-Llama-3.1-8B-Instruct-bnb-4bit" and serialize it to Ollama. This is my chat format in a file chat_format.txt:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>

{SYSTEM}<|eot_id|><|start_header_id|>user<|end_header_id|>

{INPUT}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

{OUTPUT}<|eot_id|>

Which is copy pasted from the https://colab.research.google.com/drive/1WZDi7APtQ9VsvOrQSSC5DDtxq159j8iZ?usp=sharing#scrollTo=VTzZ5oZrxkFz Ollama example notebook.

This gives me:

RuntimeError: Unsloth: Your prompt template must have 2 examples showing the user input {INPUT} and the assistant output {OUTPUT}

For example what is not allowed is just:
### Input:\n{INPUT}\n\n### Response:\n{OUTPUT}\n

What is required is 2x of this:
### Input:\n{INPUT}\n\n### Response:\n{OUTPUT}\n### Input:\n{INPUT}\n\n### Response:\n{OUTPUT}\n

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/workflow/finetuner.py", line 188, in <module>
    fire.Fire(FinetuneModel)
  File "/workflow/.venv/lib/python3.11/site-packages/fire/core.py", line 143, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/workflow/.venv/lib/python3.11/site-packages/fire/core.py", line 477, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/workflow/.venv/lib/python3.11/site-packages/fire/core.py", line 693, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/workflow/finetuner.py", line 39, in __init__
    self.load_dataset()
  File "/workflow/finetuner.py", line 96, in load_dataset
    self.dataset[key] = apply_chat_template(

So apparently the example for llama is wrong.

Google Colab
#

Unfortunately I fail to reproduce this bug in the notebook. Only happens in my system. 😦

eternal quest
#

After copy-pasting my chat template onto itself, it worked but I don't see why this doesn't happen in the notebook.

eternal quest
#

Overall I think my environment is pretty broken but I don't know why because I didn't change the code, just updated dependencies. Now I can't do a batch size of 4 when I could previously do 8. Not sure how to fix this.

#

Actually, I think the doubled prompt format doubled everything in the resulting dataset.

merry finch
#

could it be an old version of unsloth on your local machine?

#

when was the last time you updated?

eternal quest
#

These are my project dependencies:

[project]
name = "finetune-unsloth"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.11"
dependencies = [
    "datasets>=2.21.0",
    "fire>=0.6.0",
    "packaging>=24.1",
    "torch>=2.4.0",
    "unsloth[cu121-torch240]",
    "wandb>=0.17.8",
]

[tool.uv.sources]
unsloth = { git = "https://github.com/unslothai/unsloth.git" }
eternal quest
#

Please let me know if you need any additional information.

eternal quest
merry finch
eternal quest
#

Hey @merry finch sorry to necropost this. Did you have the chance to look at this and the related PR? I think it's an improvement and I'd like to contribute more to unsloth.

merry finch