vlachner/mistral-medquad-r4

Mistral 7B MedQuAD (LoRA r=4)

Training configuration

• Epochs: 1
• Batch size: 1
• Gradient accumulation: 4
• Learning rate: 2e-4
• Optimizer: paged_adamw_8bit
• Quantization: 4-bit (bitsandbytes)

Training Summary

Step	Training Loss	Validation Loss	Mean Token Accuracy
20	1.1792	1.1783	0.7006
100	0.7425	0.9143	0.7697
200	0.9692	0.8869	0.7738
300	0.9178	0.8768	0.7758
400	1.0642	0.8701	0.7773
500	0.9679	0.8673	0.7783

Final metrics

• Final Training Loss: 0.93
• Final Validation Loss: 0.87
• Final Mean Token Accuracy: 0.78
• Epochs: 1
• Total Steps: 500

Model fine-tuned on the MedQuAD dataset for medical QA using PEFT + QLoRA.