params.json is a new file added for v0.3. File is necessary for function_calling. unofficial AWQ quantized models of HF seems to haved missed the file. Is there a way to convert the params for 4bit .
{
"dim": 4096,
"n_layers": 32,
"head_dim": 128,
"hidden_dim": 14336,
"n_heads": 32,
"n_kv_heads": 8,
"norm_eps": 1e-05,
"vocab_size": 32768,
"rope_theta": 1000000.0
}
#mistralai/Mistral-7B-Instruct-v0.3 , params.json missed in unofficial Quantized
hazy jacinth
hazy jacinth
Got the full precision model to use function_calling. https://github.com/slabstech/llm-recipes/blob/main/src/mistral/local_function_call_data.ipynb . Few additional improvements from Mistral for call the tools/functions. More fine grained control is now given.
coarse reef
UserMessage(content="What's the status of my transaction T1003?")
but it calls for id t1005
hazy jacinth
UserMessage(content="What's the status of my transaction T1003?")
but it calls f...
ahh. this was due to refractor error. I just ran it again. It works fine.
this file has the correct results. Need improvment in parsing the results, we set the max_tokens to get required tool call, if large size is kept, the response is larger. I did a dirty hack to get the result.
Plus, shutdown the kernel after execution. The connection is not stop to GPU. model is still loaded in memory