onyx valve
Hi, i was recently working on a large Dataset.
Dataset is having total 20k images divided in 1k classes
and total size is 2 GB.
and while training the .h5 model using tensorflow, keras cause me a lot of RAM usage over 10 GB and which ultimately results in program crashing and my job get incompleted.
My src code for training the model will be sent here only.
if you can have a look and tell me what could be the possible reason for ram to go such high and how can i handle it.
btw i have hardware limitations of maximum of 10 GB ( which can be assigned to VS Code )
random echo
u can use aws
onyx valve
ye but it will require CC for verification which right now i can't use.
and the other thing is i guess the main reason behind the high memory is this part of code ( like it loads all images everything at once )
causing high memory usage as i have GPU in use for training purpose
onyx valve
https://sourceb.in/hgL0nYeIaS
here is the full code
i guess there maybe an alternate way to do it too ( which will increase time but ) which won't cause much RAM usage.
i just couldn't find anything related to my problem
warm yacht
first thing that comes to mind is lowering the batch size from the current 32
afterwards, if still hitting the limit, try https://www.tensorflow.org/api_docs/python/tf/data/Dataset
onyx valve
batch size isin't actually causing the issue. I already tried with keeping it to 4 too.
also a small information i forgot to mention as Windows and Tensorflow only supports GPU with tensorflow v2.10.1 so i am on that one now the latest one
onyx valve
its not good but ig good enough for training the model.
Nvidia Geforce MX570 and 551.86 driver
onyx valve
and what's the error message when you hit the ram limit?
there is no error it just automatically crashes like not responding
device: 0, name: NVIDIA GeForce MX570, pci bus id: 0000:01:00.0, compute capability: 8.6 - GPU
warm yacht
2GB gpu ram is very small, and your architecture has 22M parameters I think
onyx valve
ye true.
warm yacht
you might have to stick with cpu or offload your compute to the cloud
onyx valve
i actually tried the Google Cloud Collab too
and there also it exceeded 12 GB of normal RAM
and their GPU is bad ( free plan one ) than mine like slower in training, my 2 gb ram one is much faster than their one so i decided to train on local machine
when i try for data of like 300 classes or basically 6k images it trains it well
but when i go for 1k classes - apprx 20k images total than it crashes in b/w just due to overload in ram ( normal ram not the gpu one )
Total Size of data is 2 GB ( inc. all those 20k images )
warm yacht
I think it's possible if you try functional api where you fill up your numpy sample and labels a little bit, train on it, and repeat with the next batch. I've not tried this before though. Maybe lose the dense 128 units relu layer?
onyx valve
its always my first option to try on local machine but if no options left at end i will switch to cloud
warm yacht
cloud is a nice option if data privacy is not an issue
onyx valve
i can give a try to what you just told