Huggingface Inference Endpoints - Chat Template error

[mossy copper]

When I tried to inference in Huggingface Inference Endpoints, I got this error

{"timestamp":"2024-07-30T06:47:56.438220Z","level":"ERROR","message":"Template error: invalid operation: tried to use + operator on unsupported types string and undefined (in <string>:6)","target":"text_generation_router::infer","filename":"router/src/infer.rs","line_number":202,"span":{"name":"apply_chat_template"},"spans":[{"name":"chat_completions"},{"name":"apply_chat_template"}]}

Model - https://huggingface.co/pacozaa/mistral-sharegpt90k-merged_16bit
Here is my tokenizer_config - https://huggingface.co/pacozaa/mistral-sharegpt90k-merged_16bit/blob/main/tokenizer_config.json

[mossy copper]

Also unsloth/llama-2-7b-chat-bnb-4bit and unsloth/tinyllama-chat-bnb-4bit can't be initialized in Inference Endpoint

[mossy copper]

@noble wing Any ide on this issue? Maybe I have to write custom handler?

[tough basin]

Hey I've got it working with a custom container image - take a look here: https://github.com/kmahorker/unsloth-hf-inference

[mossy copper]

@tough basin Is it nescessary to do customer container image? Can we just use handler.py?

[tough basin]

I believe its possible, but I had issues with dependencies, since I think there's some default dependencies installed when an inference endpoint spins up - so opted for the containerized approach.

[crude gull]

Thanks for sharing this. Very interesting. Are you still using this to deploy unsloth models to HF Inference Endpoints?