Initialization of Mixtral-8x7B

[frosty topaz]

Hi, and congratulations on the Mixtral release!

My question is a simple one. From the recent a16z podcast you briefly commented on how Mixtral was made:

"You take all of the dense layers of your transformer, and you duplicate them."

I very humbly request a confirmation on whether this refers to:

• Sparse Upcycling? (https://arxiv.org/abs/2212.05055)
• Another method of initializing Mixtral-8x7B from Mistral-7B?
• Neither?

[vague forum]

I'd assume so! to paraphrase the paper, "all parameters, and optionally their optimizer state, are copied from the original checkpoint, except those corresponding to the MoE router, which does not exist in the original architecture."

[frosty topaz]

Hm, it sure does sound like it, but Aran said the method didn't scale well. If it was used to train Mixtral that would be very interesting in and of itself.