#🔒 Failed to fine tune a model with FSTP+QLoRA.

14 messages · Page 1 of 1 (latest)

sour gust Dec 1, 2024, 1:39 PM

I've trying for more then 10 hrs today, still can't figure out how to fix it.
Command to run:
accelerate launch lora_fsdp2.py

default_config.yaml

compute_environment: LOCAL_MACHINE
debug: false
distributed_type: FSDP
downcast_bf16: 'no'
enable_cpu_affinity: false
fsdp_config:
  fsdp_activation_checkpointing: false
  fsdp_auto_wrap_policy: TRANSFORMER_BASED_WRAP
  fsdp_backward_prefetch: BACKWARD_PRE
  fsdp_cpu_ram_efficient_loading: true
  fsdp_forward_prefetch: false
  fsdp_offload_params: false
  fsdp_sharding_strategy: FULL_SHARD
  fsdp_state_dict_type: SHARDED_STATE_DICT
  fsdp_sync_module_states: true
  fsdp_use_orig_params: false
machine_rank: 0
main_training_function: main
mixed_precision: fp16
num_machines: 1
num_processes: 6
rdzv_backend: static
same_network: true
tpu_env: []
tpu_use_cluster: false
tpu_use_sudo: false
use_cpu: false

lyric beaconBOT Dec 1, 2024, 1:39 PM

sour gust I've trying for more then 10 hrs today, still can't figure out how to fix it. Co...

@sour gust

Python help channel opened

Remember to:

Ask your Python question, not if you can ask or if there's an expert who can help.
Show a code sample as text (rather than a screenshot) and the error message, if you've got one.
Explain what you expect to happen and what actually happens.

:warning: Do not pip install anything that isn't related to your question, especially if asked to over DMs.

sour gust Dec 1, 2024, 1:40 PM

Code:

# accelerate launch lora_fsdp2.py
import torch
from accelerate import Accelerator
from accelerate.utils import DistributedType
from trl import SFTTrainer, SFTConfig
from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import LoraConfig, get_peft_model
# torch.set_default_device("cuda")
accelerator = Accelerator(mixed_precision="fp16")

# Load the Dataset
dataset = load_dataset("spider", split="train")  # Example: Spider text-to-SQL dataset
test_dataset = load_dataset("spider", split="validation")

# Preprocess the Dataset
def preprocess_function(examples):
    prompts = ["Generate an SQL query for the given question: " + question for question in examples["question"]]
    targets = examples["query"]
    model_inputs = tokenizer(prompts, truncation=True, padding="max_length", max_length=256)
    labels = tokenizer(targets, truncation=True, padding="max_length", max_length=256)["input_ids"]
    model_inputs["labels"] = labels
    return model_inputs


model_name = "meta-llama/Llama-3.2-1B-Instruct"
tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token
tokenized_dataset = dataset.map(preprocess_function, batched=True)
tokenized_test_dataset = test_dataset.map(preprocess_function, batched=True)

# Load the Base Model
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_storage=torch.float16,
)

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=bnb_config,
    attn_implementation="flash_attention_2",
    torch_dtype=torch.float16,
)

# Configure QLoRA with PEFT
lora_config = LoraConfig(
    r=64,
    lora_alpha=16,
    target_modules=["q_proj", "v_proj"],  # Target attention layers
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

training_args = SFTConfig(
    output_dir="./output",
    per_device_train_batch_size=6,
    per_device_eval_batch_size=6,
    gradient_accumulation_steps=16,
    learning_rate=2e-4,
    num_train_epochs=3,
    logging_dir="./logs",
    logging_steps=10,
    eval_strategy="steps",
    optim="adamw_torch",
    eval_steps=10,
    fp16=True,
    fsdp="full_shard auto_wrap",  # Fully Sharded Data Parallel
    # bf16=True,
    report_to="none",
    ddp_find_unused_parameters=False,
)


trainer = SFTTrainer(
    model=model,
    train_dataset=tokenized_dataset,
    eval_dataset=tokenized_test_dataset,
    peft_config=lora_config,
    tokenizer=tokenizer,
    args=training_args,
)

trainer.model.print_trainable_parameters()

if getattr(trainer.accelerator.state, "fsdp_plugin", None):
    from peft.utils.other import fsdp_auto_wrap_policy

    fsdp_plugin = trainer.accelerator.state.fsdp_plugin
    fsdp_plugin.auto_wrap_policy = fsdp_auto_wrap_policy(trainer.model)

trainer.model = trainer.model.to(dtype=torch.float16)
trainer.train()
# Configure FSDP for auto wrapping

# model = get_peft_model(model, lora_config)
# model = model.to(dtype=torch.float16)

# Begin Training

# Save the Model
model.save_pretrained("./finetuned-llama-text2sql")
tokenizer.save_pretrained("./finetuned-llama-text2sql")

Error messages:

[rank2]: Traceback (most recent call last):
[rank2]:   File "/workspaces/LLMTrain/lora_fsdp2.py", line 96, in <module>
[rank2]:     trainer.train()
[rank2]:   File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 2123, in train
[rank2]:     return inner_training_loop(
[rank2]:   File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 2534, in _inner_training_loop
[rank2]:     self.optimizer.step()
[rank2]:   File "/usr/local/lib/python3.10/dist-packages/accelerate/optimizer.py", line 158, in step
[rank2]:     self.scaler.step(self.optimizer, closure)
[rank2]:   File "/usr/local/lib/python3.10/dist-packages/torch/amp/grad_scaler.py", line 457, in step
[rank2]:     retval = self._maybe_opt_step(optimizer, optimizer_state, *args, **kwargs)
[rank2]:   File "/usr/local/lib/python3.10/dist-packages/torch/amp/grad_scaler.py", line 352, in _maybe_opt_step
[rank2]:     retval = optimizer.step(*args, **kwargs)
[rank2]:   File "/usr/local/lib/python3.10/dist-packages/accelerate/optimizer.py", line 203, in patched_step
[rank2]:     return method(*args, **kwargs)
[rank2]:   File "/usr/local/lib/python3.10/dist-packages/torch/optim/lr_scheduler.py", line 137, in wrapper
[rank2]:     return func.__get__(opt, opt.__class__)(*args, **kwargs)
[rank2]:   File "/usr/local/lib/python3.10/dist-packages/torch/optim/optimizer.py", line 487, in wrapper
[rank2]:     out = func(*args, **kwargs)
[rank2]:   File "/usr/local/lib/python3.10/dist-packages/torch/optim/optimizer.py", line 91, in _use_grad
[rank2]:     ret = func(self, *args, **kwargs)
[rank2]:   File "/usr/local/lib/python3.10/dist-packages/torch/optim/adamw.py", line 220, in step
[rank2]:     adamw(
[rank2]:   File "/usr/local/lib/python3.10/dist-packages/torch/optim/optimizer.py", line 154, in maybe_fallback
[rank2]:     return func(*args, **kwargs)

[rank2]:   File "/usr/local/lib/python3.10/dist-packages/torch/optim/adamw.py", line 782, in adamw
[rank2]:     func(
[rank2]:   File "/usr/local/lib/python3.10/dist-packages/torch/optim/adamw.py", line 480, in _multi_tensor_adamw
[rank2]:     grouped_tensors = Optimizer._group_tensors_by_device_and_dtype(
[rank2]:   File "/usr/local/lib/python3.10/dist-packages/torch/optim/optimizer.py", line 516, in _group_tensors_by_device_and_dtype
[rank2]:     return _group_tensors_by_device_and_dtype(tensorlistlist, with_indices)  # type: ignore[return-value, arg-type]
[rank2]:   File "/usr/local/lib/python3.10/dist-packages/torch/utils/_contextlib.py", line 116, in decorate_context
[rank2]:     return func(*args, **kwargs)
[rank2]:   File "/usr/local/lib/python3.10/dist-packages/torch/utils/_foreach_utils.py", line 37, in _group_tensors_by_device_and_dtype
[rank2]:     return torch._C._group_tensors_by_device_and_dtype(tensorlistlist, with_indices)
[rank2]: RuntimeError: Tensors of the same index must be on the same device and the same dtype except `step` tensors that can be CPU and float32/64 notwithstanding
  0%|          | 0/36 [00:14<?, ?it/s]```

GPUs:

I have no idea how to solve this:
RuntimeError: Tensors of the same index must be on the same device and the same dtype except `step` tensors that can be CPU and float32/64 notwithstanding
I thought I have everything set to fp16:
bnb_4bit_compute_dtype=torch.float16,
bnb_4bit_quant_storage=torch.float16,
torch_dtype=torch.float16,
fp16=True,

I've also tried to put torch.set_default_device("cuda") in the beginning
The error changed to:

[rank1]: Traceback (most recent call last):
[rank1]:   File "/workspaces/LLMTrain/lora_fsdp2.py", line 96, in <module>
[rank1]:     trainer.train()
[rank1]:   File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 2123, in train
[rank1]:     return inner_training_loop(
[rank1]:   File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 2427, in _inner_training_loop
[rank1]:     batch_samples, num_items_in_batch = self.get_batch_samples(epoch_iterator, num_batches)
[rank1]:   File "/usr/local/lib/python3.10/dist-packages/transformers/trainer.py", line 5045, in get_batch_samples
[rank1]:     batch_samples += [next(epoch_iterator)]
[rank1]:   File "/usr/local/lib/python3.10/dist-packages/accelerate/data_loader.py", line 552, in __iter__
[rank1]:     current_batch = next(dataloader_iter)
[rank1]:   File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 701, in __next__
[rank1]:     data = self._next_data()
[rank1]:   File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 756, in _next_data
[rank1]:     index = self._next_index()  # may raise StopIteration

[rank1]:   File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/dataloader.py", line 691, in _next_index
[rank1]:     return next(self._sampler_iter)  # may raise StopIteration
[rank1]:   File "/usr/local/lib/python3.10/dist-packages/accelerate/data_loader.py", line 214, in _iter_with_no_split
[rank1]:     for idx, batch in enumerate(self.batch_sampler):
[rank1]:   File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/sampler.py", line 347, in __iter__
[rank1]:     for idx in self.sampler:
[rank1]:   File "/usr/local/lib/python3.10/dist-packages/accelerate/data_loader.py", line 95, in __iter__
[rank1]:     yield from super().__iter__()
[rank1]:   File "/usr/local/lib/python3.10/dist-packages/torch/utils/data/sampler.py", line 197, in __iter__
[rank1]:     yield from torch.randperm(n, generator=generator).tolist()
[rank1]:   File "/usr/local/lib/python3.10/dist-packages/torch/utils/_device.py", line 106, in __torch_function__
[rank1]:     return func(*args, **kwargs)
[rank1]: RuntimeError: Expected a 'cuda' device type for generator but found 'cpu'

Which is odd. Haven't I already specified to use cuda in advanced? How do this error even happened?

sour gust Dec 1, 2024, 2:35 PM

😢

lyric beaconBOT Dec 1, 2024, 3:35 PM

@sour gust

Python help channel closed

This help channel has been closed and it's no longer possible to send messages here. If your question wasn't answered, feel free to create a new post in #1035199133436354600. To maximize your chances of getting a response, check out this guide on asking good questions.