#Batch inference broken with Gemma2

16 messages · Page 1 of 1 (latest)

low tendonBOT
#

The issue at https://github.com/unslothai/unsloth/issues/2939 describes a bug with batch inference for Gemma-2 models in Unsloth: when using a batch size greater than 1, padding causes the model to generate empty or incorrect outputs, while single-sample inference works as expected. The problem appears to be related to how padding tokens and attention masks are handled in the Gemma2 implementation. As a workaround, batching prompts of the same length (thus avoiding padding) yields correct results. The Unsloth team has acknowledged the issue, and further investigation is ongoing, but there is no official fix yet.

For more details and code samples, see the full discussion at Unsloth GitHub Issue #2939. Would you like a step-by-step breakdown or guidance on possible workarounds?

Sources:

💡 **Hint:** Mention @RunLLM in the thread for followups.
acoustic kelp
low tendonBOT
#

Sure thing! I'm looking through the available data to answer your question. Hold on a second... :hourglass_flowing_sand:

acoustic kelp
#

How to reach anyone regarding this?
Gemma2 might seem a little old but i am trying to finetune TxGemma, which is a finetune of Gemma2, so this is very relevant to me..

low tendonBOT
#

Sure thing! I'm looking through the available data to answer your question. Hold on a second... :hourglass_flowing_sand:

spice rivet
#

Will check!

acoustic kelp
# spice rivet Will check!

Any way this can be fixed?
We are currently evaluating Unsloth for Enterprise purposes and its really discouraging that something as basic as batch inference doesnt seem to work. :/

spice rivet
#

So sorry on the delay - been inundated with model releases

#

ill take a look at it today and tomorrow

acoustic kelp
#

Thanks a lot for all your work, Daniel!

spice rivet
#

as an update the main branch of unsloth has a temproary fix

#

so if u do

#

pip install --upgrade --force-reinstall --no-deps git+https://github.com/unslothai/unsloth.git

#

it should work

#

for now

#

if it doesnt work, ill see what else i can do