#MistralAPIStatusException when testing a finetuned model

28 messages · Page 1 of 1 (latest)

vernal mica
#

I finetuned a model with mistral small, and the status of the finetuning job was SUCCESS, but when i tried inference, it did this error : raise MistralAPIStatusException.from_response(response, message=str(e)) from e
mistralai.exceptions.MistralAPIStatusException: Status: 500. Message: {"object":"error","message":"Service unavailable.","type":"internal_server_error","param":null,"code":"1000"}

sonic marten
vernal mica
#

Hello, thaks for your quick response.
Here is the user id : 51c0211a-bbad-4cce-b894-a931a051f920
Here is the job id : 40fcf147-77ee-4e0d-bd54-bfc748943f4c

sonic marten
#

and the model id?

vernal mica
#

data=[Job(id='40fcf147-77ee-4e0d-bd54-bfc748943f4c', hyperparameters=TrainingParameters(training_steps=32, learning_rate=0.01), fine_tuned_model='ft:mistral-small-latest:eeb309b4:20240625:40fcf147', model='mistral-small-latest', status='SUCCESS', job_type='FT', created_at=1719322222, modified_at=1719322668, training_files=['fdb1c250-ee36-4f75-9f22-3a6af57cc41c'], validation_files=['41783589-5971-4d22-9907-fe04ff187f7b'], object='job', integrations=[])

sonic marten
#

and when you try to use ft:mistral-small-latest:eeb309b4:20240625:40fcf147 you get internal server error correct?

vernal mica
#

I did that : from mistralai.models.chat_completion import ChatMessage

from mistralai.client import MistralClient

client = MistralClient(api_key="mykey")

Retrieve a jobs

retrieved_job = client.jobs.retrieve('40fcf147-77ee-4e0d-bd54-bfc748943f4c')

chat_response = client.chat(
model=retrieved_job.fine_tuned_model,
messages=[ChatMessage(role='user', content="""

sonic marten
vernal mica
#

I tried but i get the exact same error about 30 seconds after i run the code

sonic marten
#

We will investigate ! Thanks for the report

vernal mica
#

Ok thanks !

sonic marten
# 
from mistralai.models.chat_completion import ChatMessage
from mistralai.client import MistralClient

client = MistralClient(api_key="mykey")

chat_response = client.chat(
    model="ft:mistral-small-latest:eeb309b4:20240625:40fcf147",
    messages=[ChatMessage(role='user', content="hi there")],
    max_tokens=64
)
vernal mica
#

I get the same error

sonic marten
#

we believe have found the issue, but max_tokens should at least not raise an error, thats odd... one minute!

#

could you try with this:

curl --location 'https://api.mistral.ai/v1/chat/completions' \
--header 'x-api-key: XXXX' \
--header 'Content-Type: application/json' \
--data '{
    "model": "ft:mistral-small-latest:eeb309b4:20240625:40fcf147",
    "messages": [
        {
            "role": "user",
            "content": "hello how are you"
        }
    ],
    "temperature": 0,
    "max_tokens": 10
}'
#

you can just run this curl directly

vernal mica
#

yes it seems to work but i don't know why i get infinite chineese characters 😅 Should i add an end token in the dataset or it is automatic ?

sonic marten
#

Perfect, let me explain what was happening

#

basically because you didnt set a max_tokens, it was waiting for the model to finish, but, the model was like you say only outputting chinese characters in an infinite loop- hence the internal server error

#

the issue is very likely your learning rate

#

0.01 is too high

#

dont go higher than 0.001

#

anyways, when testing the model always set a max tokens to be sure it doesnt loop

vernal mica
#

Thx, you are a life saver. And so setting max tokens with mistralai client is not working ?

sonic marten
#

would you mind reinstalling mistralai and try again with max_tokens to see if it still happens? might be an issue we are not aware of