ERNIE 4 5 VL 28B A3B PT Vision
unsloth-notebooksunslothoriginal_template
Export
News
Placeholder
Installation
[ ]
Unsloth
[ ]
🦥 Unsloth: Will patch your computer to enable 2x faster free finetuning. 🦥 Unsloth Zoo will now patch everything to make training faster!
A new version of the following files was downloaded from https://huggingface.co/unsloth/ERNIE-4.5-VL-28B-A3B-PT: - configuration_ernie4_5_vl.py . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
Unsloth: WARNING `trust_remote_code` is True. Are you certain you want to do remote code execution? ==((====))== Unsloth 2025.11.3: Fast Dfnrope_Vision_Transformer patching. Transformers: 4.56.2. \\ /| NVIDIA A100-SXM4-80GB. Num GPUs = 1. Max memory: 79.318 GB. Platform: Linux. O^O/ \_/ \ Torch: 2.9.0+cu126. CUDA: 8.0. CUDA Toolkit: 12.6. Triton: 3.5.0 \ / Bfloat16 = TRUE. FA [Xformers = 0.0.33.post1. FA2 = False] "-____-" Free license: http://github.com/unslothai/unsloth Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored! Unsloth: QLoRA and full finetuning all not selected. Switching to 16bit LoRA.
modeling_ernie4_5_vl.py: 0.00B [00:00, ?B/s]
A new version of the following files was downloaded from https://huggingface.co/unsloth/ERNIE-4.5-VL-28B-A3B-PT: - modeling_ernie4_5_vl.py . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
model.safetensors.index.json: 0.00B [00:00, ?B/s]
model-00001-of-00012.safetensors: 0%| | 0.00/4.99G [00:00<?, ?B/s]
model-00002-of-00012.safetensors: 0%| | 0.00/4.99G [00:00<?, ?B/s]
model-00003-of-00012.safetensors: 0%| | 0.00/5.00G [00:00<?, ?B/s]
model-00004-of-00012.safetensors: 0%| | 0.00/5.00G [00:00<?, ?B/s]
model-00005-of-00012.safetensors: 0%| | 0.00/4.99G [00:00<?, ?B/s]
model-00006-of-00012.safetensors: 0%| | 0.00/5.00G [00:00<?, ?B/s]
model-00007-of-00012.safetensors: 0%| | 0.00/5.00G [00:00<?, ?B/s]
model-00008-of-00012.safetensors: 0%| | 0.00/5.00G [00:00<?, ?B/s]
model-00009-of-00012.safetensors: 0%| | 0.00/5.00G [00:00<?, ?B/s]
model-00010-of-00012.safetensors: 0%| | 0.00/4.99G [00:00<?, ?B/s]
model-00011-of-00012.safetensors: 0%| | 0.00/5.00G [00:00<?, ?B/s]
model-00012-of-00012.safetensors: 0%| | 0.00/3.88G [00:00<?, ?B/s]
Loading checkpoint shards: 0%| | 0/12 [00:00<?, ?it/s]
generation_config.json: 0%| | 0.00/228 [00:00<?, ?B/s]
tokenizer_config.json: 0.00B [00:00, ?B/s]
The repository unsloth/ERNIE-4.5-VL-28B-A3B-PT contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co/unsloth/ERNIE-4.5-VL-28B-A3B-PT . You can inspect the repository content at https://hf.co/unsloth/ERNIE-4.5-VL-28B-A3B-PT. You can avoid this prompt in future by passing the argument `trust_remote_code=True`. Do you wish to run the custom code? [y/N] y
processing_ernie4_5_vl.py: 0.00B [00:00, ?B/s]
A new version of the following files was downloaded from https://huggingface.co/unsloth/ERNIE-4.5-VL-28B-A3B-PT: - processing_ernie4_5_vl.py . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
tokenizer.model: 0%| | 0.00/1.61M [00:00<?, ?B/s]
added_tokens.json: 0.00B [00:00, ?B/s]
special_tokens_map.json: 0.00B [00:00, ?B/s]
chat_template.jinja: 0.00B [00:00, ?B/s]
We now load the processor
[ ]
processor_config.json: 0%| | 0.00/308 [00:00<?, ?B/s]
chat_template.json: 0.00B [00:00, ?B/s]
preprocessor_config.json: 0%| | 0.00/823 [00:00<?, ?B/s]
We now add LoRA adapters for parameter efficient finetuning - this allows us to only efficiently train 1% of all parameters.
[NEW] We also support finetuning ONLY the vision part of the model, or ONLY the language part. Or you can select both! You can also select to finetune the attention or the MLP layers!
[ ]
Unsloth: Making `model.base_model.model.model` require gradients
[ ]
README.md: 0%| | 0.00/519 [00:00<?, ?B/s]
data/train-00000-of-00001.parquet: 0%| | 0.00/344M [00:00<?, ?B/s]
data/test-00000-of-00001.parquet: 0%| | 0.00/38.2M [00:00<?, ?B/s]
Generating train split: 0%| | 0/68686 [00:00<?, ? examples/s]
Generating test split: 0%| | 0/7632 [00:00<?, ? examples/s]
Let's take an overview look at the dataset. We shall see what the 3rd image is, and what caption it had.
[ ]
Dataset({
, features: ['image', 'text'],
, num_rows: 68686
,}) [ ]
[ ]
'H ^ { \\prime } = \\beta N \\int d \\lambda \\biggl \\{ \\frac { 1 } { 2 \\beta ^ { 2 } N ^ { 2 } } \\partial _ { \\lambda } \\zeta ^ { \\dagger } \\partial _ { \\lambda } \\zeta + V ( \\lambda ) \\zeta ^ { \\dagger } \\zeta \\biggr \\} \\ .' We can also render the LaTeX in the browser directly!
[ ]
$\displaystyle H ^ { \prime } = \beta N \int d \lambda \biggl \{ \frac { 1 } { 2 \beta ^ { 2 } N ^ { 2 } } \partial _ { \lambda } \zeta ^ { \dagger } \partial _ { \lambda } \zeta + V ( \lambda ) \zeta ^ { \dagger } \zeta \biggr \} \ .$To format the dataset, all vision finetuning tasks should be formatted as follows:
[
{ "role": "user",
"content": [{"type": "text", "text": Q}, {"type": "image", "image": image} ]
},
{ "role": "assistant",
"content": [{"type": "text", "text": A} ]
},
]
[ ]
Let's convert the dataset into the "correct" format for finetuning:
[ ]
We look at how the conversations are structured for the first example:
[ ]
{'messages': [{'role': 'user',
, 'content': [{'type': 'text',
, 'text': 'Write the LaTeX representation for this image.'},
, {'type': 'image',
, 'image': <PIL.PngImagePlugin.PngImageFile image mode=RGB size=160x40>}]},
, {'role': 'assistant',
, 'content': [{'type': 'text',
, 'text': '{ \\frac { N } { M } } \\in { \\bf Z } , { \\frac { M } { P } } \\in { \\bf Z } , { \\frac { P } { Q } } \\in { \\bf Z }'}],
, 'reasoning_content': '\n'}]} Let's first see before we do any finetuning what the model outputs for the first example!
[ ]
The LaTeX code for the Hamiltonian in the image is:
H'=\beta N\int d\lambda\left\{\frac{1}{2\beta^{2}N^{2}}\partial_{\xi}\lambda^{\dagger}\partial_{\xi}\lambda+V(\lambda)\zeta^{\dagger}\zeta\right\}~.</s>
[ ]
[ ]
[ ]
GPU = NVIDIA A100-SXM4-80GB. Max memory = 79.318 GB. 56.051 GB of memory reserved.
[ ]
==((====))== Unsloth - 2x faster free finetuning | Num GPUs used = 1 \\ /| Num examples = 68,686 | Num Epochs = 1 | Total steps = 30 O^O/ \_/ \ Batch size per device = 2 | Gradient accumulation steps = 2 \ / Data Parallel GPUs = 1 | Total batch size (2 x 2 x 1) = 4 "-____-" Trainable parameters = 308,224,000 of 29,707,518,336 (1.04% trained) Could not estimate the number of tokens of the input, floating-point operations will not be computed
[ ]
755.0562 seconds used for training. 12.58 minutes used for training. Peak reserved memory = 62.695 GB. Peak reserved memory for training = 6.644 GB. Peak reserved memory % of max memory = 79.043 %. Peak reserved memory for training % of max memory = 8.376 %.
Inference
Let's run the model! You can change the instruction and input - leave the output blank!
We use min_p = 0.1 and temperature = 1.5. Read this Tweet for more information on why.
[ ]
H ^ { \prime } = \beta N \int d \lambda \left\{ \frac { 1 } { 2 \beta ^ { 2 } N ^ { 2 } } \partial _ { \lambda } \zeta ^ { \dagger } \partial _ { \lambda } \zeta + V ( \lambda ) \zeta ^ { \dagger } \zeta \right\} \ .<|end_of_sentence|>
[ ]
('lora_model/tokenizer_config.json',
, 'lora_model/special_tokens_map.json',
, 'lora_model/chat_template.jinja',
, 'lora_model/tokenizer.model',
, 'lora_model/added_tokens.json') Now if you want to load the LoRA adapters we just saved for inference, set False to True:
[ ]
Saving to float16 for VLLM
We also support saving to float16 directly. Select merged_16bit for float16. Use push_to_hub_merged to upload to your Hugging Face account! You can go to https://huggingface.co/settings/tokens for your personal tokens.
[ ]