Run multiple finetuning on same GPU POD | Runpod | Page 1

red atlas Jun 14, 2024, 10:55 AM

#

I am using

image: runpod/pytorch:2.2.0-py3.10-cuda12.1.1-devel-ubuntu22.04
GPU: 1 x A40

While running qlora finetuning with 4 bit quantization the GPU uses approx 12 GB GPU Memory out of 48 GB, how can I run multiple finetunings simultaneously (in parallel) on the same POD GPU?

tribal spruceBOT Jun 14, 2024, 10:55 AM

#

To help others find answers, you can mark your question as solved via Right click solution message -> Apps -> ✅ Mark Solution

arctic imp Jun 14, 2024, 11:05 AM

#

Depends on your application you're using.

red atlas Jun 14, 2024, 11:09 AM

#

Okay but how? I am using python for running the finetunings

hexed pulsar Jun 14, 2024, 11:14 AM

#

Hmm... Okay how do you connect to the gpu then?

#

What framework do you use?

#

And search it on google

red atlas Jun 14, 2024, 11:51 AM

#

I have a script that is has the address to model, tokens, output directory and dataset.

Lets say I manually run it once and the finetuning starts
Then I change the values of output dir and dataset to perform another finetuning

Will the POD GPU be able to handle it properly?

red atlas Jun 14, 2024, 11:51 AM

#

hexed pulsar What framework do you use?

huggingface, torch and transformers

hexed pulsar Jun 14, 2024, 12:01 PM

#

red atlas I have a script that is has the address to model, tokens, output directory and d...

Sure why not? Depends on the available resources..

#

If it has enough then it will run smoothly

#Run multiple finetuning on same GPU POD