Deepseek OCR (3B) Evaluation
unsloth-notebooksunslothoriginal_template
Export
News
Placeholder
Installation
[ ]
Unsloth
Let's prepare the OCR model to our local first
[ ]
[ ]
🦥 Unsloth: Will patch your computer to enable 2x faster free finetuning. 🦥 Unsloth Zoo will now patch everything to make training faster!
You are using a model of type deepseek_vl_v2 to instantiate a model of type DeepseekOCR. This is not supported for all configurations of models and can yield errors.
Unsloth: WARNING `trust_remote_code` is True. Are you certain you want to do remote code execution? ==((====))== Unsloth 2025.11.1: Fast Deepseekocr patching. Transformers: 4.56.2. \\ /| NVIDIA A100-SXM4-40GB. Num GPUs = 1. Max memory: 39.557 GB. Platform: Linux. O^O/ \_/ \ Torch: 2.8.0+cu126. CUDA: 8.0. CUDA Toolkit: 12.6. Triton: 3.4.0 \ / Bfloat16 = TRUE. FA [Xformers = 0.0.32.post2. FA2 = False] "-____-" Free license: http://github.com/unslothai/unsloth Unsloth: Fast downloading is enabled - ignore downloading bars which are red colored!
You are using a model of type deepseek_vl_v2 to instantiate a model of type DeepseekOCR. This is not supported for all configurations of models and can yield errors.
Unsloth: QLoRA and full finetuning all not selected. Switching to 16bit LoRA.
Some weights of DeepseekOCRForCausalLM were not initialized from the model checkpoint at ./deepseek_ocr and are newly initialized: ['model.vision_model.embeddings.position_ids'] You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
[ ]
Let's Evaluate Deepseek-OCR Baseline Performance on Persian Transcription
[ ]
Loading evaluation dataset...
README.md: 0%| | 0.00/967 [00:00<?, ?B/s]
data/train-00000-of-00002.parquet: 0%| | 0.00/255M [00:00<?, ?B/s]
data/train-00001-of-00002.parquet: 0%| | 0.00/256M [00:00<?, ?B/s]
data/test-00000-of-00001.parquet: 0%| | 0.00/57.1M [00:00<?, ?B/s]
Generating train split: 0%| | 0/179999 [00:00<?, ? examples/s]
Generating test split: 0%| | 0/20000 [00:00<?, ? examples/s]
Running Baseline Evaluation...
Evaluating: 100%|██████████| 200/200 [03:12<00:00, 1.04it/s]
============================================================
Baseline Model Performance
============================================================
Number of samples: 200
Mean CER: 149.07%
Median CER: 80.00%
Std Dev: 310.39%
Min CER: 0.00%
Max CER: 3500.00%
============================================================
Best Predictions (Lowest CER):
Sample 5024 (CER: 0.00%)
Reference: چون هستی خیلی زیاد...
Prediction: چون هستی خیلی زیاد...
Sample 3517 (CER: 0.00%)
Reference: تو ایران هیچوقت از اینها وجود نخواهد داشت...
Prediction: تو ایران هیچوقت از اینها وجود نخواهد داشت...
Sample 9949 (CER: 0.00%)
Reference: کاش میدونستم هیچی بیخیال...
Prediction: کاش میدونستم هیچی بیخیال...
Worst Predictions (Highest CER):
Sample 11155 (CER: 3500.00%)
Reference: خسو...
Prediction: \[ \text{CH}_3\text{CH}_2\text{CH}_2\text{CH}_2\text{CH}_2\text{CH}_2\text{CH}_2\text{CH}_2\text{CH}...
Sample 13366 (CER: 1900.00%)
Reference: مشو...
Prediction: \[\begin{align*}\underline{\mathfrak{su}}_0\end{align*}\]...
Sample 10552 (CER: 1014.29%)
Reference: هیییییچ...
Prediction: e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
e
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o
o...
✅ Results saved to baseline_evaluation.json
Mean Baseline Model Performance: 149.07% Character Error Rate (CER) for this eval set !
Let's finetune Deepseek-OCR !
We now add LoRA adapters for parameter efficient finetuning - this allows us to only efficiently train 1% of all parameters.
[NEW] We also support finetuning ONLY the vision part of the model, or ONLY the language part. Or you can select both! You can also select to finetune the attention or the MLP layers!
[ ]
Unsloth: Making `model.base_model.model.model` require gradients
Data Prep
We'll be using a dataset for Persian OCR. The goal is to convert these images into a computer readable form - ie text. This can be very useful for digitizing Persian text.
You can access the dataset here.
Let's take an overview look at the dataset. We shall see what the 3rd image is, and what caption it had.
To format the dataset, all vision finetuning tasks should be formatted as follows:
[
{ "role": "<|User|>",
"content": "",
"images": []
},
{ "role": "<|Assistant|>",
"content": ""
},
]
[ ]
Let's convert the dataset into the "correct" format for finetuning:
[ ]
We look at how the conversations are structured for the first example:
[ ]
{'messages': [{'role': '<|User|>',
, 'content': '<image>\nFree OCR. ',
, 'images': [<PIL.JpegImagePlugin.JpegImageFile image mode=RGB size=218x48>]},
, {'role': '<|Assistant|>', 'content': 'همهاش جبره و اختیار توهمه'}]} [ ]
[ ]
/tmp/ipython-input-4060472171.py:12: FutureWarning: `tokenizer` is deprecated and will be removed in version 5.0.0 for `Trainer.__init__`. Use `processing_class` instead. trainer = Trainer(
[ ]
GPU = NVIDIA L4. Max memory = 22.161 GB. 8.012 GB of memory reserved.
[ ]
==((====))== Unsloth - 2x faster free finetuning | Num GPUs used = 1 \\ /| Num examples = 1,000 | Num Epochs = 1 | Total steps = 60 O^O/ \_/ \ Batch size per device = 2 | Gradient accumulation steps = 4 \ / Data Parallel GPUs = 1 | Total batch size (2 x 4 x 1) = 8 "-____-" Trainable parameters = 77,509,632 of 3,413,615,872 (2.27% trained) Unsloth: Not an error, but DeepseekOCRForCausalLM does not accept `num_items_in_batch`. Using gradient accumulation will be very slightly less accurate. Read more on gradient accumulation issues here: https://unsloth.ai/blog/gradient
Unsloth: Will smartly offload gradients to save VRAM!
You are using a model of type deepseek_vl_v2 to instantiate a model of type DeepseekOCR. This is not supported for all configurations of models and can yield errors.
Now after fine-tuning lets evaluate the model!
[ ]
Evaluating: 100%|██████████| 200/200 [06:11<00:00, 1.86s/it]
============================================================ Fine-tuned Model Performance ============================================================ Number of samples: 200 Mean CER: 60.43% Median CER: 50.00% Std Dev: 80.63% Min CER: 0.00% Max CER: 916.67% ============================================================ Best Predictions (Lowest CER): Sample 301 (CER: 0.00%) Reference: باشه بابا تو لاکچری، تو خاص، تو خفن... Prediction: باشه بابا تو لاکچری، تو خاص، تو خفن... Sample 2512 (CER: 0.00%) Reference: از شخص حاج عبدالله زنجبیلی میگیرنش... Prediction: از شخص حاج عبدالله زنجبیلی میگیرنش... Sample 2713 (CER: 0.00%) Reference: نمی دونم والا تحمل نقد ندارن ظاهرا... Prediction: نمی دونم والا تحمل نقد ندارن ظاهرا... Worst Predictions (Highest CER): Sample 14270 (CER: 916.67%) Reference: ۴۳۵۹۴۷۴۷۳۸۹۰... Prediction: پروپریپریپریپریپریپریپریپریپریپریپریپریپریپریپریپریپریپریپیپریپریپریپریپریپریپریپریپریپریپریپریپریپر... Sample 3919 (CER: 380.00%) Reference: ۷۵۵۰۷۱۰۶۵۹... Prediction: وادووووووووووووووووووووووووووووووووووو... Sample 3718 (CER: 333.33%) Reference: ۳۲۶۷۲۲۶۵۵۸۴۶... Prediction: پُپُسوپُسوپُسوپُسوپُسوپُسوپُسوپُسوپُسوپُ... ✅ Results saved to finetuned_evaluation.json
Now lets compare both.
[ ]
============================================================ 📈 PERFORMANCE COMPARISON ============================================================ Baseline Mean CER: 149.07% Fine-tuned Mean CER: 60.43% ✨ Absolute Improvement: 88.64% ✨ Relative Improvement: 59.46% ============================================================
With only 60 steps, we reduced the Character Error Rate (CER) from 149.07% to 60.43%, representing a 88.6% absolute improvement in character errors!
[ ]
[ ]
پاکستان ===============save results:===============
image: 0it [00:00, ?it/s] other: 0it [00:00, ?it/s]
[ ]
You are using a model of type deepseek_vl_v2 to instantiate a model of type DeepseekOCR. This is not supported for all configurations of models and can yield errors.
('lora_model/tokenizer_config.json',
, 'lora_model/special_tokens_map.json',
, 'lora_model/tokenizer.json') Now if you want to load the LoRA adapters we just saved for inference, set False to True:
[ ]
Saving to float16 for VLLM
We also support saving to float16 directly. Select merged_16bit for float16. Use push_to_hub_merged to upload to your Hugging Face account! You can go to https://huggingface.co/settings/tokens for your personal tokens.
[ ]
And we're done! If you have any questions on Unsloth, we have a Discord channel! If you find any bugs or want to keep updated with the latest LLM stuff, or need help, join projects etc, feel free to join our Discord!
Some other resources:
- Train your own reasoning model - Llama GRPO notebook Free Colab
- Saving finetunes to Ollama. Free notebook
- Llama 3.2 Vision finetuning - Radiography use case. Free Colab
- See notebooks for DPO, ORPO, Continued pretraining, conversational finetuning and more on our documentation!
