#Training LLM in a specific language

1 messages · Page 1 of 1 (latest)

upbeat forge
#

Hello everyone, I am new to LLMs, although I have previous expertise in software programming. I'd like to learn how to train an LLM in a specific language (English, German, French).

  • Should I utilize a pre-trained model (Falcon / Dolly) and fine-tune using the new language data set? Is there a way to block the other languages?
  • Should I train a model from scratch? If so does this mean I need to create a model from scratch or can I use one that is open-source?
sonic gulch
#

Training model from scratch is too expensive.

Finetuning won't be effective for learning a new language IMO.

Use a multilingual LLM like BLOOM.

upbeat forge
#

Hi @sonic gulch, I have checked a few multilingual LLMs but they aren't very efficient in the language I am interested in. Another problem that I have found with the multilingual LLMs is that they are heavy compared to LLMs that are trained in only one language. Why wouldn't finetuning be effective for learning a new language?

sonic gulch
gusty pebble
#

I think this will help answer some of your questions, also do you have enough data? Understanding how transformers work would help too in terms of embedding, positional encoding, multi-head attention etc

https://www.youtube.com/watch?v=3XiJrn_8F9Q&ab_channel=KrishNaik

YouTube
Krish Naik
Custom Training Question Answer Model Using Transformer BERT

Simpletransformer library is based on the Transformers library by HuggingFace. Simple Transformers lets you quickly train and evaluate Transformer models. Only 3 lines of code are needed to initialize a model, train the model, and evaluate a model.
github: https://github.com/krishnaik06/Trnasformer-Bert
simple transformer: https://simpletransfor...

▶ Play video