train_stsb_101112_1760638037

This model is a fine-tuned version of meta-llama/Meta-Llama-3-8B-Instruct on the stsb dataset. It achieves the following results on the evaluation set:

• Loss: 0.4393
• Num Input Tokens Seen: 8712528

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.03
• train_batch_size: 4
• eval_batch_size: 4
• seed: 101112
• optimizer: Use adamw_torch with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: cosine
• lr_scheduler_warmup_ratio: 0.1
• num_epochs: 20

Training results

Training Loss	Epoch	Step	Validation Loss	Input Tokens Seen
0.6419	1.0	1294	0.4976	434624
0.402	2.0	2588	0.4681	869056
0.3758	3.0	3882	0.4575	1304160
0.4497	4.0	5176	0.4449	1740416
0.3365	5.0	6470	0.4393	2175568
0.4103	6.0	7764	0.4405	2611168
0.4	7.0	9058	0.4405	3047200
0.4053	8.0	10352	0.4508	3482720
0.3003	9.0	11646	0.4752	3918416
0.4131	10.0	12940	0.4874	4355072
0.4063	11.0	14234	0.4960	4790336
0.3632	12.0	15528	0.5173	5227040
0.2523	13.0	16822	0.5918	5662848
0.3145	14.0	18116	0.5935	6099600
0.2143	15.0	19410	0.6925	6534256
0.2152	16.0	20704	0.7844	6968992
0.1456	17.0	21998	0.9381	7405040
0.1341	18.0	23292	1.0220	7840784
0.0741	19.0	24586	1.0400	8276160
0.3049	20.0	25880	1.0398	8712528

Framework versions

• PEFT 0.17.1
• Transformers 4.51.3
• Pytorch 2.9.0+cu128
• Datasets 4.0.0
• Tokenizers 0.21.4