Mistral V0.3 (7B) CPT
News
Placeholder
Installation
Unsloth
We now add LoRA adapters so we only need to update 1 to 10% of all parameters!
We also add embed_tokens and lm_head to allow the model to learn out of distribution data.
Data Prep
We now use the Korean subset of the Wikipedia dataset to first continually pretrain the model. You can use any language you like! Go to Wikipedia's List of Languages to find your own language!
[NOTE] To train only on completions (ignoring the user's input) read TRL's docs here.
[NOTE] Remember to add the EOS_TOKEN to the tokenized output!! Otherwise you'll get infinite generations!
If you want to use the llama-3 template for ShareGPT datasets, try our conversational notebook
For text completions like novel writing, try this notebook.
[NOTE] Use https://translate.google.com to translate from English to Korean!
We only use 1% of the dataset to speed things up! Use more for longer runs!
Continued Pretraining
Now let's use Unsloth's UnslothTrainer! More docs here: TRL SFT docs. We do 20 steps to speed things up, but you can set num_train_epochs=1 for a full run, and turn off max_steps=None.
Also set embedding_learning_rate to be a learning rate at least 2x or 10x smaller than learning_rate to make continual pretraining work!
Instruction Finetuning
We now use the Alpaca in GPT4 Dataset but translated in Korean!
Go to vicgalle/alpaca-gpt4 for the original GPT4 dataset for Alpaca or MultilingualSIFT project for other translations of the Alpaca dataset.
We print 1 example:
We again use https://translate.google.com/ to translate the Alpaca format into Korean
We again employ UnslothTrainer and do instruction finetuning!
Inference
Let's run the model! You can change the instruction and input - leave the output blank!
Remember to use https://translate.google.com/!
You can also use a TextStreamer for continuous inference - so you can see the generation token by token, instead of waiting the whole time!
By using https://translate.google.com/ we get
Korean music is classified into many types of music genres.
This genre is classified into different music genres such as pop songs,
rock songs, classical songs and pop songs, music groups consisting of drums, fans, instruments and singers
Now if you want to load the LoRA adapters we just saved for inference, set False to True:
By using https://translate.google.com/ we get
Earth refers to all things including natural disasters such as local derailment
and local depletion that occur in one space along with the suppression of water, gases, and living things.
Most of the Earth's water comes from oceans, atmospheric water, underground water layers, and rivers and rivers.
Yikes the language model is a bit whacky! Change the temperature and using sampling will definitely make the output much better!
You can also use Hugging Face's AutoModelForPeftCausalLM. Only use this if you do not have unsloth installed. It can be hopelessly slow, since 4bit model downloading is not supported, and Unsloth's inference is 2x faster.
Saving to float16 for VLLM
We also support saving to float16 directly. Select merged_16bit for float16 or merged_4bit for int4. We also allow lora adapters as a fallback. Use push_to_hub_merged to upload to your Hugging Face account! You can go to https://huggingface.co/settings/tokens for your personal tokens.
GGUF / llama.cpp Conversion
To save to GGUF / llama.cpp, we support it natively now! We clone llama.cpp and we default save it to q8_0. We allow all methods like q4_k_m. Use save_pretrained_gguf for local saving and push_to_hub_gguf for uploading to HF.
Some supported quant methods (full list on our Wiki page):
q8_0- Fast conversion. High resource use, but generally acceptable.q4_k_m- Recommended. Uses Q6_K for half of the attention.wv and feed_forward.w2 tensors, else Q4_K.q5_k_m- Recommended. Uses Q6_K for half of the attention.wv and feed_forward.w2 tensors, else Q5_K.