Weird Mixtral 8x7B Output | Mistral AI | Page 1

frank cosmos Dec 21, 2023, 2:47 PM

#

Hi, I am using the quantized Mixtral 8x7B version from "TheBloke" (Q_5_0; Q_5_K_M). Generally I am happy with the generations for my RAG task but for some specific question, the generation takes very long and the output does not makes sense (mostly something like "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n..."). What could be a reason for this?

I set n_ctx to 28000
I am working on a MacBook Pro with 12 croes and 64 GB memory.
I am following the specific Prompt format

tulip hazel Dec 21, 2023, 4:53 PM

#

EDITED: PEBCAK

velvet steppe Dec 21, 2023, 4:56 PM

#

Use instruct version

tulip hazel Dec 21, 2023, 4:58 PM

#

velvet steppe Use instruct version

Thanks, that does give much more expected results. Didn't realize how little the base model had been fine tuned.

#Weird Mixtral 8x7B Output